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First Graders, Hypermedia and Creativity 



MARYL. STEPHEN 
Department of Computer Science and Computer Education 
Harris-Stowe State College 

3026 Laclede Avenue 
St. Louis, MO., USA, 63103 

In 1989 a partnership was established between Harris-Stowe State College, a historically black teacher 
education institution, and Waring Academy of Basic Instruction, a magnet school in the St. Louis Public 
Schools. In 1992, the computer education instructor at the college and a first grade teacher at the magnet 
school initiated a project to introduce a hypermedia package to the first graders and to explore ways to 
integrate use of the program into the first grade curriculum. Only three of the 23 students in the original group 
of first graders had access to computers outside of the school setting. The project included use of the one 
computer in the students* classroom and weekly visits to a computer lab at the college. The original goals of 
the project were to determine whether first grade students could learn to use a hypermedia authoring system in 
a meaningful way, to determine if using such a program enhanced the first grade curriculum, and to determine 
any benefits from using such a software package with first graders. 

The results of the first year of this project were extremely encouraging. The students mastered the software 
package much more quickly than anticipated. Students learned to create stacks which incorporated original 
artwork, text, clip art, original sounds, clip sounds, visible and invisible buttons, card transitions and text 
items. Lack of initial reading skills did not hinder the students* ability to learn the package. Students were 
taught keyboard commands for the basic actions, thus eliminating the need to read items in menus. Curricular 
areas which appeared to be positively impacted by this project include reading, language arts, math, science, 
social studies and art. In addition, the level of cooperative learning was outstanding. Students often referred to 
their computer experiences to help them solve problems in the more traditional classroom setting. The first 
grade teacher, a 33 year veteran, indicated that she had never observed such a degree of self-confidence 
among first graders before. The teacher also noted that there appeared to be a definite transfer effect in the 
students* sequencing, reading, spelling, writing and fine motor skills, as well as creativity level. 

The project is currently in the second year, with a new class of first graders now participating. The 
students involved this year share many of the characteristics noted previously in the original class of students. 
Students have already learned more features of the package than students involved in the first year of the 
project, a fact the teachers involved attribute to their initial fear that they would overwhelm the students if they 
presented too many features. Observations from the previous year have convinced them otherwise. Once again 
all students involved exhibit the ability to work well cooperatively, and the same high level of self confidence 
observed in the students in the first year of the project. 
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Kids get the good stuff: A progress report on Jostens Learning's 
networked interactive multi-media mathematics product 

Christopher W. Cook and Roy B. Clariana 

Jostens Learning 
7878 N 16th St, Phoenix, AZ 85020 



Jostens Interactive Multi-Media (IMM) is a k-6 curriculum in mathematics and language 
arts utilizing full-motion interactive video delivered over a local area network. Units include 
individualized and small group activities both on-line and off-line, and on-line teacher 
demonstration and presentation activities. The software will be delivered in stages. The first 
portion to be released in the summer of 1994 is called "Teacher First" and consists of the 
teacher presentation and demonstration activities in mathematics. We will present several of 
these activities today and allow time after the session for you to have hands-on with the demo 
unit. Because of the limited time available, most of the presentation will focus on Teacher 
First, though we will address some issues regarding IMM. 

Teacher First was designed to be a low-cost, entry level product. It runs stand alone 
utilizing either MS-DOS or Mac machines. The MS-DOS machine requires at least a 
486sx/33MHz, 12MB RAM (16 MB for Capture/Edit), 80 MB hard drive (1 GB for 
Capture/Edit), CD-ROM, ActionMedia II Card, VGA monitor, DSP 301 Audio/Ethernet 
card, and CS2 Capture Card. The Mac machine must be either an LCIII or Quadra, 12MB 
RAM (16 MB for Capture/Edit), 80 MB hard drive (1 GB for Capture/Edit), CD-ROM, and 
New Video EYE-Q video card. Teacher First software can also run over the Jostens 
Learning Multi-Media File Server. 

The instructional design is a direct implementation of the NCTM standards and utilizes 
methodological constructivist ideas. Specifically, the activities address mathematics concepts 
within theme-based units or contexts (rather than as isolated skills). The software is thus 
only one component of the total learning environment. Accompanying instructional materials 
include a teachers 1 guide with detailed lesson plans. A typical unit involves "center" activities 
where kids work together on assigned tasks in small groups. Centers include manipulatives, 
games, and other types of activities. 

Ease of use is a critical instructional issue when using multi-media applications in the 
classroom. The software interface is a unique and simple part of Teacher First. Alpha 
product testing has shown that even naive mathematics educators can utilize the interface 
optimally after even a brief exposure. 

To summarize, Teacher First brings together all of the current threads in mathematics 
education into one complete, easy to use application. This may be an evolutionary product 
that changes the way teachers teach as well as the way students will learn. 
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Putting the "E" in NKEN: 
Implications of Connecting K-12 Schools to the 
Information Superhighway 



Linda M. Baker 
Educational Communications and Technology 
University of Washington, Miller Hall DQ-12, Seattle, Washington USA 
lbaker@u. Washington, edu 

The National Research and Information Network (NREN), the "information superhighway M proposed by national 
legislation, holds great promise for K-12 schools. Yet there are major technical, economic and structural obstacles 
to overcome to realize that promise, and often unexamined legal, cultural and social impacts of connecting schools. 
The methods of technology assessment can be used to systematically explore the obstacles and impacts so pohcy 
decisions can be made which will help K-12 schools determine what is desirable, as well as realize what is possible. 

The presentation considered three major questions related to school networking, borrowed from McClure, 
Bishop, Doty and Rosenblum (1991). 

(I) . How can the use of electronic networks facilitate the tasks and goals of 

particular communities of users? 

(II) . What problems do particular communities of users face in attempting to 

exploit networks for accomplishment of those tasks and goals? 

(III) . What design, management and policy strategies can alleviate those problems 

and maximize network use and effectiveness? 
These three questions were addressed considering the special needs of K-12 educators and students as the community 
of users. Extensions of NREN to the K-12 community means that an 'elite* medium of scientific communication 
and collaboration becomes more a 'mass medium' by connecting many more users of a diverse kind. It also means 
connection of children, a protcted group with special developmental needs, to a largely adult medium. What 
implications do these changes have for schools? 

Why technology assessment? 

Technology assessment (TA) arose as a field of study in the 1960s and 70s to help systematically consider the 
impacts adoption of a particular technology might have for society as a whole before the technology was widely 
adopted. Environmental impact assessments are the most widely recognized and often practiced form of technology 
assessments. 

Not many professionals interested in educational technology are aware of the formal methods of technology 
.assessment, yet some of these methods could be productively used to address educational technology issues and give 
focus to policy debates. The presentation introduced participants briefly to a few of the techniques of TA to illustrate 
how TA can help answer some major questions in our field, as well as raise different kinds of questions not always 
considered in the rush to adopt whatever is newest. If K-12 educators wish to influence national telecommunication 
policy, such as NREN, then it would be helpful to know in which directions their best interests lie, something TA 
can help them determine. TA can also outline new areas for research, as the process often points out areas where 
empirically-derived information is either lacking or inadequate. 

Participants were given a four page bibliography on technology assessment, its methods and relevant studies. 
References 

McClure, Bishop, Doty, & Rosenblum. (1991) The National Research and Information Network (NREN): Research 
and Policy Perspectives. Norwood, New Jersey: Ablex Publishing. 
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GeoSim: A GIS-Based Simulation Laboratory for 
Introductory Geography 

Clifford A. Shaffer*, James M.A. Begole*, Laurence W. Carstensen Jr.**, 
Robert W. Morrill** and Edward A. Fox* 
Department of Computer Science* and Department of Geography** 
Virginia Polytechnic Institute and State University 
Blacksburg, VA 24061 
email: geosim@cs.vt.edu 



We demonstrate three modules currently under development for Project GeoSim, a multidisci- 
plinary effort by members of Virginia Tech's Departments of Geography and Computer Science, College 
of Education, and Learning Resources Center to develop computer-aided education (CAE) software for 
introductory geography and related classes. GeoSim modules are designed to teach students the concepts 
of dynamic geographic processes through interactive exploration. The modules combine the information 
presentation and analysis capabilities of Geographic Information Systems (GIS) with interactive tech- 
niques of computer simulation. Through a multidisciplinary approach aimed solely at educational needs, 
we automate the use of GIS and simulation so a full understanding of spatial and statistical analysis 
techniques is not a prerequisite to learning from them. 

GeoSim modules will become part of existing comprehensive courses without relying on instruc- 
tors to modify their courses to "fit them in" or worse to buy expensive equipment on which to operate 
them. The modules fit closely into existing models of introductory geography as illustrated by current 
popular texts. The modules will run on IBM compatible 80x86 machines, Macintosh II computers, and 
UNIX systems running X-Windows. 

All GeoSim modules share the s?,me menu-driven interface. In general, the modules begin with 
a multimedia tutorial presenting the information to be learned, followed by a simulation to allow the 
student to actively use the information. 

The three modules to be presented are as follows. 

International Population allows students to investigate the effects of altered birth, death, and 
net migration rates on the population pyramids and total population levels for any country or region 
in the world. The simulation supports comparison of two countries simultaneously, and permits three 
different scenarios to be displayed at one time for each country. 

Modeling of Migration in the United States relates county migration patterns in the U.S. 
between 1950-1990 with data on place characteristics. Students will be asked to select and weight push 
and pull factors to model migration patterns. They will learn about the scientific method by making 
hypotheses regarding effects on migration patterns and then immediately testing these hypotheses. 

Mental Maps is a simple graphical quiz program that tests students* knowledge of the location 
and characteristics of cities by having the student point to a position on an outline map drawn on the 
computer screen and then answer a series of questions. After the student selects city placements, a "mental 
map" showing the student's perception of the geography of the country is generated and displayed. 

GeoSim modules are available through anonymous FTP or gopher at "geosim.cs.vt.edu." 
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The Cari D. Perkins Vocational Education Act mandates that technology education be included in home 
economics instruction. New curriculum is being introduced into the home economics classroom to help 
secondary students understand the interdependence of technology and family life. To prepare future teachers for 
these curriculum changes, all home economics education students at the University of Arkansas are required to 
become familiar with general and educational uses of the computer and learn how to create an awareness of how 
new technological innovations affect family life. 

Students are required to use word processing, data base and spreadsheet applications as well as home 
economics courseware. For teacher assistance, they use word processing to write lesson plans and professional 
resumes, data bases to develop test question banks for multiple choice tests, organize inventories and maintain 
student records, and spreadsheets to plan budgets and determine grades. Courseware is integrated into nutrition, 
consumer education, housing and clothing classes. For classroom applications, students learn to incorporate 
tutorials and simulation software into lessons, use nutritional analysis software to analyze the adequacy of 
diets, use design software to plan spaces and garments, as well as use application software to help learners 
explore personal and family finance. Draw software is used to encourage students' creativity. Desktop 
publishing is used to create flyers and brochures for the student member section of the American Home 
Economics Association. With the assistance of the technology education instructor, students are introduced to 
robots and their use in home and industry. 

Students plan technology-related lessons using activities in Impact of Technology on the Family 
(Segraves, Carrick, Pence, Brixcy, James, Knight, Gregory & Kellner, 1992). Some examples of lesson 
objectives are evaluating technology used by children, determining technology to solve family problems, 
evaluating how technology will affect future jobs, evaluating the ethics of the use of technology in the medical 
field, and designing technologically based products for the home. Students also evaluate technology used in 
household equipment such as microwaves, small kitchen appliances, home computers, video games, stereos, 
televisions and photo compact discs. 

Home economics education students receive a minimum of twenty hours of direct computer instruction in 
four home economics education classes as well as individual assistance with personal and class projects. 
Students also plan computer related projects during students teaching using equipment found in the secondary 
schools. 

Students work with Macintosh, IBM, and Apple lie computers to prepare them to use the equipment they 
will find in secondary classrooms and computer laboratories. They are also introduced to CD ROM and 
interactive video technology. 



Segraves, D., Carrick, M M Pence, D., Brixey, B., James, C, Knight, R., Gregory, L. and Kellner, P. (1992). 
Impact of Technology on the Family. Stillwater, OK: Mid-America Vocational Curriculum Consortium, 



Technology and the Family: Incorporating Technology 
Education into Home Economics Education 



Dr. Cecelia Thompson and Dr. Dale E. Thompson 
Department of Vocational and Adult Education 
University of Arkansas, 210 Home Economics Building 
Fayetteville, AR 72701 
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Impact Evaluation of a CBT Program Developed to Increase 
the Effectiveness of Law Enforcement and Prevent Police Civil Liability 

Mohammad Aj.am 
Police Law Institute 
Technology Innovation Center, Room 225 
Oakdale, Iowa 52319 



Abstract: A CBT project has immediate and long-term objectives. Immediate 
objectives are related to the mastery of the learning material and this can be 
accomplished within the time-frame of the training program. Long-term goals are 
related to the ultimate impact of the CBT program and cannot be accomplished until 
some time after the completion of the training program. This paper examines the 
long-term impact of a CBT project developed to increase the effectiveness of law 
enforcement and prevent police civil liability. 

Losses from police civil liability cases averaged $800 per officer in 1987. An effective way to prevent 
court judgments for civil liability is to prevent illegal mistakes by the officers on the street through continuous 
training and testing of officers on the correct legal way to do the high risk tasks of police work. 

MIRMA is a non-profit self-insurance pool that covers 60 cities in Missouri. It insures the towns and 
municipal utilities against all claims made as a result of the official actions by all their employees. About 16% 
of the employees of the member governments are police (1250 officers.) Throughout the early to mid 1980s, 
losses for claims of misconduct by police had escalated at the national rate. In 1 989, MIRMA instituted a 
proactive loss preventing program for police activities. As part of this program, MIRMA contracted with the 
Police Law Institute to provide monthly computerized leg^l training to all sworn personnel of the 60 member 
municipalities. 

Within a year, the impact of this continuous training became apparent. While the national insurance rates 
for police liability have continued to increase, the cost of liability for MIRMA cities has declined: (1) The losses 
have declined each year that the training program has been in effect.; (2) From fiscal years 1989 to 1991, the 
cost of losses, on a per-officer basis, had declined 66%. The cost of losses for 1989 were three times those of 
1991; and (3) When the effects of price inflation, the reduction is even more dramatic. MIRMA's projections 
show that without the loss prevention program, the per-officer losses in 1991 would have been five times greater 
than the amount that they actually incurred. 

One of the other long-term objectives of the Police Law Institute's computer-based training program is to 
increase the effectiveness of police work. This implies more successful conviction of criminals and reduced 
crime rate. These are two long-term goals that have not been evaluated yet, partly because of the difficulty in 
gathering the necessary relevant data. 
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New Strategies for Learning and Performance Assessment of 
Interpersonal Problem-Solving Skills 



J. Olin Campbell and Cheryl A. Lison 
Corporate Learning Institute, Peabody College of Vanderbilt University 
Peabody Box 321, Nashville, TN 37203 
E-Mail: campbejo@ctrvax.vanderbilt.edu 

Abstract: In a world of \ ^bal competition, the ability to collaborate 
to solve problems is crit. *: 1. However, learning such skills can be 
difficult and expensive. ^ rategies that use simulations can be helpful 
for * .hnical training, but we iack strong evidence of the effects of 
similar methods for interpersonal problem solving. This paper 
discusses two experimental studies that address these issues. The 
studies used both hard (computer and video) and soft (instruction 
design) technologies. Experiment 1 kept the instructional methods as 
similar as possible, but cut instructor time by 50 %, by substituting 
computer tutoring and skill modeling on video for corresponding 
methods in the classroom. The computer and video group performed 
as well as the classroom group. Experiment 2 cut instructor time by 
67% using similar substitutions, and also optimized the methods, 
using video recording and small group feedback on individual 
performance. It found significant improvement of subjects' 
performance in the optimized group over the classroom group. 

We live in a world where global competition places a premium on collaborating on 
problem solving. Working with another person to solve a problem is one of the most 
critical skills to master and yet one of the most expensive to instruct, because it requires 
expert modeling and coaching. The limited availability and high costs of experts limits 
access to such skills. Computer-based instruction by itself is of limited help because 
problem solving interactions are ill defined. In addition, computer hardware and software 
has very limited ability to understand spoken discourse and intonation, or to interpret visual 
cues such as body posture. We must work out combinations of instructor, students, and 
technology to deal with these issues. 

Much of our current instruction yields what Whitehead (1929) terms inert 
knowledge that is not used when the ne^d arises. Learners must learn how to perform in 
complex situations, and also how to learn on their own to perform. Performance-based 
simulation is a powerful strategy for situating learning in contexts that are like those 
where learners will use the skills. By controlling the complexity and adding coaching and 
evaluative feedback, the simulation may become a learning situation and an assessment 
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tool. Using similar coaching, the system may also become a job aid/performance support 
tool. 



Problem: Cost 



There is a problem, however: cost. A method that uses lecture to "cover the 
content," followed by multiple choice tests is typically far less expensive to develop and 
deliver to large numbers of people simultaneously than is individually coached and 
assessed performance development. The cost is lower, although the benefits may not be 
as great. 

There are several possible responses to this issue that may either increase 
productivity or decrease costs: 

1 . Use computer and other simulations that successively increase the level of challenge to 
the learner (e.g., micro-worlds and the graduated length ski method discussed by 
Burton, Brown, and Fischer, 1984) 

2. Use evaluative feedback in the simulations (Gibbons & Rogers, 1991) 

3. Use the simulation for both learning and assessment, thus sharing the costs (e.g., 
Frederiksen & White, 1990; Bunderson, Olsen, & Greenberg, 1992; Campbell, 1992) 

4. Use the simulation for performance support when the skills are used (Campbell, 1992) 

5. Use computer-aided instructional design tools (e.g., Pirolli, 1991) 

Most of the prior work on learning simulations has been for technical skills on 
complex, expensive, or dangerous equipment. In these cases the cost savings can be very 
high in relation to purchasing actual equipment for training purposes (for example, an 
airplane, a telephone exchange, or a radar system). 

In this paper we discuss the extension of simulations having some of the above 
properties to interpersonal skills. We note that these simulations use students, the 
instructor, computer, and video. We also note that performance support may come at low 
cost if it is already part of the coaching component of an evaluative simulation. 

The Problem Solving With Others Study 

There is some promising research in the area of interactive training for "soft skills" 
Cronin & Cronin (1992). The research methodology of the original studies was often 
questionable, however, given the difficulty with assessing interpersonal skills. To extend 
the line of research, our group conducted two studies, each with a two-group experimental 
design. 

The first study contrasted a classroom lecture/discussion/small group method with 
a similar method that used computer and video. We used the computer and video vehicle 
where possible to help students learn certain skills and to see the skills modeled. By doing 
so we cut the amount of instructor time in half for the same level of performance, but kept 
the methods as similar as possible. 
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In the second study we optimized the combined instaictor/computer/video 
treatment, in which small teams practiced together. For both groups we created a scoring 
instrument that would capture a circumscribed set of skills related to facilitating problem 
solving by another person. We have validated the interpretation of this instrument by 
using outside blind raters, who did not know what treatments were used or who was 
assigned to each treatment. 

A classroom treatment used lecture-discussion, modeling, role play, and small team 
interactions, with the instructor facilitating each activity. A combined instructor and 
technology treatment used computer tutorials and modeling on videotape to supplement the 
instructor's similar learning activities. 

Both treatments required nine hours of subject time in the first experiment and six 
hours in the second experiment. Subjects were tested in a videotaped role-play. In the 
combined instructor and technology treatment we decreased instructor time by 50% in 
Experiment I, and by 67% in Experiment 2. In Experiment 1, our analysis indicated that 
the combined group scored as well on the posttest as the classroom group. In Experiment 2, 
the optimized group showed significant gains, but the classroom group did not. An 
independent rating by three counseling students, who were blind to the experimental 
treatments, also found a significant difference on the posttest favoring the group using the 
combined treatment. 

This research has borne fruit: We found ways to use small teams of learners with 
relatively low technology simulation tools (e.g., computer tutorial and videotape modeling 
of skills) to yield significant time savings and increased student performance over the 
traditional classroom method. This work is reported by Campbell, Lison, Borsook, 
Hoover, and Arnold (in press). From the studies noted above, we find support for the 
hypothesis that treatments using technology -supported methods can decrease the amount 
of instructor time needed to learn interpersonal problem solving skills, while achieving 
results comparable to a fully instructor-led learning experience using similar methods. 
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Abstract: Within Mathematics of Finance classes at the Smeal College of Business 
Administration at Penn State University, lectures arc developed using Asymetric's 
Toolbook and are presented via a computer system. This approach was implemented 
because it has the potential to effectively convey concepts that arc ordinarily difficult to 
communicate to large classes of students. For example, the class can input arbitrary 
interest rates and instantly see their effect on other variables. 

In the fall semester of 1992, an analysis was conducted on a class of 58 junior level 
Actuarial Science students. The methods used to make this analysis were to video record 
all class lectures with two video cameras. One camera focused on the students and 
collected their physical reactions such as facial expressions and body movements and the 
second camera recorded images projected by the computer system. Students also received 
a pre and post attitude survey to collect their perception of the computer-assisted 
lectures. 



The Smeal College of Business Administration at the Penn State University continues to investigate 
the potential of computer-assisted learning systems. Within Mathematics of Finance classes, lectures are 
developed using Asymetrix Toolbook and are presented via a computer system. This approach was 
implemented because it has the potential to effectively convey concepts that are ordinarily difficult to 
communicate tc large classes of students. For example, arbitrary interest rates can be fed to the system and 
students can instantly see a visual representation of their effect on other variables. The instructor can ask 
questions and input student responses and electronically graph the result during lectures to help monitor the 
classes' attentiveness or understanding of the material. This approach adds additional dynamics to lectures and 
assignments. 

While it seemed, from an instructor perspective, that the computer-assisted learning approach could 
positively impact classes, it was unclear as to how students would react to it A pilot study was therefore 
conducted to collect student perceptions and reactions to this instructional approach. The objectives were to 
make an initial assessment of the computer-assisted lecture delivery format; to collect student reactions to 
this delivery format; and to investigate the potential of video recording as a method of data collection 
because, among other things, videotaping has been used extensively for subject observation with substantial 
benefits reported (Baum & Gray, 1992). 
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Methods 

In the fall semester of 1992, an analysis was conducted on a class of 58 junior level Actuarial Science 
students. In previous years, the general characteristics of the classroom were as follows: each student had a 
copy of the professor's typed notes upon which they wrote comments; the professor knew each student by 
name; an attempt was made to call upon each student at least once each class to respond to question related 
material; and an important part of each lecture was explaining concer o the "man on the street" The 
change this semester was that lectures were presented via a computer systems using Toolbook. 

There were two methods of analysis used. One method was to video record all class lectures. For 
video recording, the researchers used two cameras. One camera focused on students to collect their physical 
reactions such as facial expressions and body movements and a second camera recorded the visual images 
projected by the computer system. The researchers later combined the video signal so that physical 
reactions could be seen simultaneously with what was projected electronically by the computer. Thirty-two 
75 minute class lectures were recorded of which 7 are examined in this paper. For analysis, non-verbal 
communications as well as classroom interactions were examined relative to the delivery format. 

The second method used to assess the impact of this technology was an attitude survey. In week one 
of class, students were given an introduction to computer-assisted lectures and informed that throughout the 
course, lectures of this type would be given. At the end of the first week, students received a pre-attitude 
survey . 

A post-attitude survey, identical in nature to the pre-attitude survey, was given in the last week of 
class. The researchers compared surveys to observe if any differences occurred in attitude toward the computer- 
assisted lectures from the beginning of the semester to the end. 

Each survey presented statements to which respondents rated their reactions on a 9 point scale from 0 • 
to 8. The scale ranged from Strongly Agree (0 points), to Indifferent (4 points), to Strongly Disagree (8 
points). Of the 58 students who completed a pre-attitude survey, only 31 completed a post-attitude survey. 
The analysis of survey data is, therefore, based on 31 respondents. 

Results & Discussion 

Videotaping method 

Video recording class lectures appears to have potential merit in monitoring student reactions to new 
technological approaches in the classroom. The video medium provided a permanent record of students and 
how they responded to the computer-assisted lectures. Numerous observations could be made specifically 
about the technology, as well as the class in general, and these observations could be reviewed repeatedly and 
verified by others. 

Video recording lends itself to the collection of large amounts of various kinds of data, which requires 
more Lime, effort, and resources to analyze. The advantages of this technique relative to cost need to be more 
thoroughly investigated. At present, however, the video recording approach as presented in this study appears 
to be an effective method for identifying student reactions to computer-assisted lectures. 

Videotaping results: clas sroom interaction 

For this analysis, tapes from five class sessions were reviewed. The researchers look at the following 
classroom interactions: 

• areas to which the majority students directed their attention (e.g., professor's typed 

notes, the professor or the projected image) in response to the computerized lectures; 
areas to which the majority students directed their attention in responses to type of 
visual images projected (e.g., graphs). 

Computerized lectures can, in many cases, take significantly longer to develop than traditional 
lectures. Because of this time and labor intensiveness, it is important to ensure that the visual information 
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projected effectively cues students and is useful to them. By making observations, the researcher hoped to 
improve the presentation material by identifying screens designs which caught student attention and designs 
to which students lesponded favorably. 

To determine whether or not screen designs effectively cued student attention, the researchers counted 
each time the majority of students look up or looked down in response to what was projected by the 
computer. Four categories were created: 



1 . No visual and looking up 

The number of times the majority of students were looking up when no visual was projected 

by the computer. 

2. No visual and looking down 

The number of times the majority of students were looking down when no visual was 
projected by the computer. 

3 . Visual on and looking up 

The number of times the majority of students were looking up when a visual was projected 

by the computer. 

4 . Visual on and looking down 

The number of times the majority of students were looking down when a visual was 
projected by the computer. 

If the purpose of presenting visual information via a computer system is to promote, in some way, 
student understanding, then it becomes important to ensure that the visual effectively conveys its meaning 
and that students attend to it. The primary reason, therefore, for identifying these categories was to 
determine, in a general sense, if students viewed the visual information when it was presented. The highest 
incidences occurred in the visual on and looking down category (see Table 1). 



Table 1 

Response to Visuals: Attention Areas 





Class 1 


Class 2 


Class 3 


Class 4 


Class 5 


Total 


No visual 

& looking 
up 


15 


0 


12 


1 


3 


31 


No visual 
& looking 
down 


10 


0 


7 


13 


0 


30 


Visual on 
<& looking 
up 


9 


15 


2 


19 


13 


58 


Visual on 
& looking 
down 


16 


14 


8 


22 


IB 


78 

i 



This means the there were 78 incidences when the majority of students were looking down for the period of 
time when the visual was projected. It should be pointed out that students had the professor's typed notes 
with which to follow the lecture. Thus, it is to be expected that they would page through these notes while 
the concept was discussed. Oftentimes the professor talked through visuals using the computer mouse to 
point out key areas and formulas. Frequently during this time, a large portion of the class referred to the 
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professor's notes and paged hack and forth through them. It appeared that, among other things, the notes 
enabled students to bring together additional information that may or may not have beer presented on screen. 
The reliance on the notes seems to provide an additional means by which students individually verified the 
presented information and or came to understand it 

Given the value of the projecting visual information for the entire class to view and the way in which 
students, during class lectures, utilized the professor's typed notes, it is difficult to assess ihe effectiveness of 
screen designs. A future inquiry might include a control group which is presented the visual information 
unaccompanied by the professor's notes to more accurately identify student reactions. 

Videotaping results: reactions to visual type* 

The researchers wanted to identify the types of visuals (text, animated graphs or tables) which most 
effectively cued student attention. It appeared that students looked down or took notes when visuals 
containing text and formulas were presented. On most occasions, however, when the professor presented 
graphs containing animations, students looked up. Two reasons are suggested this observation. First, the 
professor's typed notes contained formulas and replicated what was present on screen and thus there may have 
been no need for students to look up. They could easily follow the professor by looking at his typed notes. 
Second, unlike text and formulas, animated tables and graphs can not be duplicated on paper. Animation 
may have received more attention because of its distinctiveness. Animation as an attention gaining device 
can be effective since attention is influenced by unique stimuli. Possibly, animating tables and graphs was 
unique enough to gain students' attention. 

General observations: 

Video recording made additional observations about the class possible and some of these observations 
are presented below. 

For five of the class sessions, the number of student initiated interactions with the professor 
were recorded. Each time a student commented or asked a question, it was considered an 
interaction. If a student, for example, asked a question and then followed up immediately 
with a second question, then it was counted as one interaction. There were 83 occurrences 
when students initiated interaction with tte* professor, 32 of which were initiated by males 
and 51 by females. Such information may prove useful to the professor since classroom 
interaction among all students was highly encouraged. 

Cueing students with key words is attention getting. Students, as observed in this study, 
spent much time following along with paper handouts and often ignored what was projected 
on screen. For example, when the professor said, "I have a picture here." students looked up, 
and on subsequent occasions the word "picture" appeared to get their attention. 

Much can be gained from observing students and watching their body language and reactions. 
In some cases an observer can readily tell when students do not understand or are confused 
(Alessi & Trollip, 1985). Throughout the course the professor constantly challenged 
students by posing questions. Students often displayed behaviors like smiling or looking 
downward when they did not know the answer. It was also clear by viewing the video tapes 
when more than one student was having difficultly with a concept or question. In such cases 
students looked to one another, shook their head or made a facial expression. 

Attitude Survey Results 

On a survey, subjects ranked the number of computer courses tl j had completed on a 10-point scale 
ranging from 0 to 9 or more completed courses. Completed computer courses was used as a measure of 
computer knowledge (Rattanapian, 1992). The researchers obtained this measure to determine if knowledge 
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about computers influenced perceptions of the lectures. On average, students completed 2 computer courses. 
Figure 1 shows the percentages of respondents by number of courses completed. 

Courses Completed 




80 100 



% Responding 
i : gure 1. Computer Courses Completed 

Given the proliferation of technology in today's classrooms and the use of computers to present 
information electronically, the researchers sought to identify the extent to which students in this class had 
been exposed to computerized lectures. On the pre-survey, students ranked their experience and on average 
reported having no experiences with the lecture format, (see figure 2). 

Experience Level 



Experienced (3+ courses) 



Used it 



Have seen it 



Have heard about it 




No experience 



10 20 30 40 



50 



% Responding 



Figure 2. Experience with Computer-Assisted Lectures 

Despite the lack of exposure, students anticipated that the computer would not make lectures more 
complex (see figure 3). On the survey, for example, students were asked if they thought the computerized 
lectures would be easy to follow. Responses to this item were positive and this perception remained 
constant from the pre to the post survey. 
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% Responding 
50 




Figure 3* Lectures were easy to follow pre and post surveys 



Students anticipated that the lectures would be informative (see figure 4). On the survey, students • 
were presented the following item, "Computer-assisted lectures will be informative." Responses tended to be 
on the positive side and this perception remained constant throughout the semester. 



% Responding 




Strongly Agree Strongly Disagree 



Figure 4. Lectures were informative pre and post surveys 

Computer familiarity and attitude 

Classroom computers are neutral in design but lack neutrality often times in the way they arc used 
(Jones, 1987) and perceived. Males, for example, have been found to readily express interest in computers 
while females are less likely to do so (Miura, 1987). Students in this study had varying levels of computer 
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expertise ranging from novice to experienced users. The researchers were interested in determining if 
computer familiarity effected (positively or negatively) students' attitude toward lectures. As a measure of 
familiarity, number of computer courses completed was used based on the assumption that some familiarity 
with computers is required with course completed. Familiarity did not appear to negatively bias students' 
opinion of lectures and in fact, students appeared to look forward to the teaching format 

At the beginning of the semester, in the pre-attitude survey, students were asked if they thought the 
lectures would dramatically impact the way in which the course was taught Responses, examined in terms 
of computer familiarity, show that for some respondents the perceived degree of impact of the computerized 
lectures decreased as the number of computer courses completed increased (see figure 5). Thus, those having 
completed more computer courses felt that the computerized lectures would have less of an impact than those 
with fewer courses. This may suggest that those with more courses completed understand the technology and 
its limitations to a greater extent than those with fewer courses. It should be pointed out, however, that only 
7 students completed more than 3 courses and the majority of respondents completed fewer than 2 courses. 
From such a small sample, it is difficult to assess whether or not this trend would persist for larger 
populations. 

Number of Courses 




012345678 
Strongly Impact No Impact 

Average Rating by Category 

Figure 5. Perceived Impact by Number of Computer Courses 

On the pre-attitude survey, students indicated whether or not they would feel comfortable being taught 
with computerized lectures. For this analysis, there were 15 out of the 32 respondents who had completed 
fewer than 3 computer courses and they indicated that they would feel comfortable being taught with 
computerized lectures (see figure 6). Of these 15 respondents, fewer computer courses completed coincided 
with favorable ratings. There appeared to be a tendency for those who had completed fewer courses to 
indicate that they would feel comfortable being taught with computerized lectures. Again, because of the 
lack of distribution of students across the different levels of courses completed, it should not be concluded 
that those with more familiarity would feel uncomfortable being taught with computerized lectures. On the 
post-survey, attitudes toward being comfortable with computerized lectures seemed to even out 
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0 1 2 
Strongly Impact 

Average Rating by Category 

Figure 6. Comfort by Computer Courses completed: pre survey 



Animations and attitude 

Many of the lectures included animations of tables and graphs to visually represent concepts and ideas. 
The professor could, for example, when discussing interest rates, ask students to suggest an appropriate rate 
of interest. The givcin rate could be input into the computer and by animating the graph or table effects on 
other variables could be instantly represented. Students perceived the animation of concepts positively both 
prior to their exposure to animation and after long-term exposure to it From the data collected students 
anticipated at the beginning of the semester that animations of graphs would be attention getting and useful 
(see figure 7). This perception remained constant, for the most part, from the beginning of the semester to 
the end. 

% Responding 
50 



-a- Pre 
-a- Post 




012345678 
Strongly Agree Strongly Disagree 



Figure 7. Animating Graphs: pre and post survey 
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Summary 

This pilot study attempted to make an initial assessment of student reactions to computer-assisted 
lectures and thus no efforts was made to control for extraneous factors which may have influenced student 
attitudes during the course of the semester. Given the constraints of this study, the researchers hesitate to 
make any generalizable conclusions, however, some interesting observations were made. First, video 
recording class lecuires with two video cameras is an effective technique to assess the implementation of a 
new classroom technology. Video preserves rich details about observations which can be studied repeatedly 
and validated by other researchers (Van Dalcn, 1979). One drawback to this technique is that video recording 
lends itself to the collection of large amounts of data and consequently requires substantial resources in time, 
money and energy to make thorough analyses. Second, upon review of the video recordings it appeared that 
students tended not to look at visuals which included text or formulas and appeared more attentive to 
animated tables and graphs. This was expected because students had the professor's typed notes with which 
to follow the lectures. Third, the subject population had very little computer knowledge or exposure to 
computer-assisted lectures. Results, however, from the pre-survey show that students anticipated that the 
lectures would be easy to follow and informative and that they would feel comfortable being taught with this 
delivery format The researchers were concerned that students, without much computer familiarity, might be 
skeptical about the delivery format which may impact learning. Fourth, students appeared to perceive 
animation of concepts and ideas as useful and attention gaining. Moreover, its use in this study whereby the 
professor maintained an interactive dialogue with students and fed their responses into the computer and 
animated results, from the researchers' perspective, added additional dynamics to class lectures and 
assignments. This technique also helped to monitor student attentiveness in a large lecture room. 
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Abstract: Computer graphics are often claimed to provide substantial motivation impact on 
learning computer-assisted instruction. While considerable development effort has focused on 
designing computer graphics, more attention needs to be placed on motivational variables that 
contribute to the effective interaction between the graphics and the learners in order to deliver the 
important information in the instruction. An integration of Keller's ARCS model ot 
motivational design (1991) is used in this paper to address the issues of attention directing, 
relevance matching, comprehension guiding , and satisfaction from the graphics in order to 
enhance the motivational elements of instrucdon. Specific guidelines are proposed based on the 
literature. 

Introduction 

Computer graphics have been widely used in today's computer-assisted instruction (CAI) I and >nteract,ve 
learning maSs« a strategy to motivate learning and to deliver important messages. With the advent of modern 
S ologT various compute? generated graphics software programs have been developed to enable graphic artists 
to design quality graphics (Alesandrini, 1987). New technology provides opportunities to make the ^signed 
messafes more appealing o viewers. However, more attention needs to be paid to design issues, such as whether 
^SS^SSSS^ to target learners or whether the graphics used are relevant to the context to be 

^Tueasy to dazzle learners with attractive graphics in the computer or interactive learning environment 
without havTng much impact on what or how well things are learned (Dam, 1992). Graphics hemselves will not 
provide any knowledge acquisition unless people who employ them provide appropriate links to the learning 
Context. Written for instructional designers, this paper discusses design issues tor graphics based on 'the 
motivational conceptual framework of Keller (1991). Incorporaung findings and implicauons from the ^mre 
T^phic use des gn variables are carefully considered. In addition to focusing on the visual aspects of graphics, 
Z^SSSS^i the motivational factors related to effective interaction between perceptions of graphics 
and learner characterisdes. 

Motivation Factors 

Motivating learning is an important issue that has been too litde explored in research and theory (Re.geluth, 
1993) Researchers have recognized that different motivations inrluence achievement (Mizelle, Hart, & Can. 
1993) However, how instruction should be designed in order to encourage students involvement in learning 
requires further exploration. "More work is needed in this area, especially regarding mot.vauonal strategies which 
are uniquely possible with advanced technologies" (Reigeluth, 1993, pp. 53). nn „:A orina 

Accordine to Keller (1987, 1990), the motivational quality of instructional materials is affected by considering 
the major strategies for attention, relevance, confidence, and satisfaction. For enhancing the motivational aspects 
of Kraphics in a CBI lesson, it is essential to understand how each individual varies in processing graphics and how 
graphics can be provided through the instructional materials in order to support perception, understanding, and 
memorization (Cate, 1993; West, 1992). Since there exist differences among individuals in organizing, 
transforming, evaluating, processing, and encoding ability (Paivio, 1990), it is essential to take into account 
Serteteristics, such as their gender, age, cognitive ability, and their knowledge background is order to meet 
the purpose of \^ designed graphics. . . 

Applied in B .uphics design, Keller's ARCS model is used to identify ways that graphics can be provided to 
fulfill motivational requirements. The model is a problem solving model and also a model used by instructional 
SgnTrs or instructors to analyze motivational requirements (Visser, 1990). In this P^r, the ARCS model is 
used as a motivational framework to derive strategies for designing computer graphics (Table 1). In the following 
sections how these motivational strategies can be extracted and applied in designing computer graphics is 
discussed in detail. These strategies prescribe the requirements for graphics design with an aim to provide a 
motivational learning environment in computer-based materials. 
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Attention 

Since computer-assisted instructional materials are usually employed for self-paced learning, getting and sustaining 
learners' attention is an essential prerequisite to involvement in the learning task (Keller & Burkman, 1993; Keller 
& Suzuki, 1988). The use of graphics in instruction, especially in computer-assisted instruction, is an 
instructional strategy that merits attention (Alesandrini, 1987); affective and emotional reactions are more easily to 
be activated with images than words (see review by Paivio, 1990). The capability of graphics for attaining and 
sustaining attention is derived from the psychological effect they provide for perception arousal. The stages 
involved in perceiving information involves receiving any given information as stimuli through sensors and 
cognitively selecting and filtering the elements among given stimuli. Attention of learners needs to be "uided 
since the perception of an image varies depending on their previous experience, mood, and personal interests 
(Pettersson, 1989). 

Used without precaution, graphics might harm learning. Presumably, the use of computer graphics in CAI 
materials is to motivate learning by guiding learners to important messages. However, irrelevant graphics 
contribute little in motivation or might even distract learners from acquiring important information (Alesandrini 
1985). Lack of potential links between personal interest and context presented might make the graphic materials 
confusing to the intended learners. It's worth further examination about how to effectively employ graphics to 
facilitate learners 1 perception arousal, interest matching and learning curiosity from the given information Several 
strategies for attention getting are described in the following section. 

Perception Arousal for Getting Attention 

It is observed that motion creates potential learning interest during interaction with computer-assisted 
materials (ChanLin, 1993; Rieber, 1990). With the integration of new technology, such as animation and visual 
effects, graphics can more easily affect the perception of given stimuli and provide learning interest and promote 
incidental learning from the instructional materials (Alesandrini, 1987; Rieber, 1989a; Rieber, 1990) Due to the 
physical differences in the ways information is presented, the use of these design techniques offers potential visual 
contrast over the static background to catch viewer's eyes and emotion (Hannafin & Peck, 1988). Use of cueing 
devices, such as color, labels, and arrows can aid discrimination or facilitate the identification of essential" learning 
characteristics in the visualized message (Beck, 1987). The instructional stimuli received by the senses must be 
adequately coded prior to being transmitted and processed (Lamberski & Dwyer, 1983). Since pictorial materials 
often contain large numbers of stimuli, the learners may need assistance in locating the most relevant information 
If learners do not receive sufficient direction for coding the essential part of the information, learning will likely 
become less than optimum. 

Inquiry Arousal for n etting Attention 

When various design strategies are employed in gaining learners attention, instructional designers should move 
their focus from facilitating sensor-level reaction (perceptual curiosity) to information seeking or problem solving 
reaction (epistemic curiosity) (Keller, 1983). Understanding the environment surrounding the learners and trying 
to integrate the events or objects that are relevant to their learning interest is more likely to encourage learners' 
involvement and their curiosity in information seeking. Matching design objects with contemporary events or 
learners current interests creates substantial curiosity when searching specific in formation (Alesandrini 1985) To 
attract and maintain curiosity, the learning materials should not only give people the opportunity to learn more 
about things they already know about or believe in, but also give them moderate doses of the unfamiliar and 
unexpected (Keller & Burkman, 1993). V . selection of graphic objects requires designers' creativity in creating an 
interesting learning environment. The inclusion of unusual and exotic materials can help viewers initiate or 
maintain curiosity. 

Variability for Getting AMfiMififl 

The use of various graphics is a way to prevent learners from feeling bored, and at the same time, to provide 
opportunity for learners to visualize different interesting themes for interaction (Hannifin & Peck, 1988- 
OConnell, 1992). A novelty approach encourages learner's curiosity in exploring interests (Keller, 1983- Keller 
& Burkman, 1993). With inclusion of diverse visualization, learners are invited to continue seeing novel' themes 
in the program. The use of different attention getting treatments in the graphics also encourages learner's curiosity 
in exploring new interests and in gaining new insights from the learning materials (Dam, 1992) It was observed 
that the aesthetics of graphical materials stimulate people's emotional feelings that can command attention * 
( Sn ?«n ^ P f Conne ."' 1 L " 2) ' The use of multi-dimensional representations of a graphic also allows effective 
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v since 

Designing graphics for effective communication needs to account for how learners make predictions and 
evaluations from the information perceived (Reynolds & Baker, 1987). Prior to designing graphics, it is important 
to consider what might impel people to learn, and determine what might stimulate people s desire in learning from 
the materials provided, and then design the materials to meet those motivational purposes. The important 
implications of successfully employing graphics in facilitating interest, comprehension and retention of the 
messages is not only grounded upon whether those pictures contribute to reflecting the textual messages but also 
how well the message is designed to be perceived, understood, and remembered by the learners in the way expected 

(SCh Thrmeaning of pictures is not inherent but is the result of constructive processes that a learner brings to bear 
on the message (Schallert, 1980). Learners exhibit strong tendencies to interpret and remember visualization in 
terms of their existing knowledge. Based on the interaction between the pictorial message and students 
understanding about The area they are studying, they translate the pictorial message with their own meaning ; and 
then reconstruct their existing knowledge by integrating the translated meaning from the message. Under this 
mplication. selecting graphics to reflect the messages that guide a learner's interpretive and constructive process 
from what they already know and what they are familiar with is important. Strategies for increasing the relevant 
of the lesson for the learners, such as goal orientation, motive matching and relating fam.lianty are discussed in 
the following section. 

Goal Orienta 'inn for Providing Relevance . ...... 

Goal orientation is considered by Keller as a value judgement for learners to link the new learning with their 
perception of importance (Keller, 1983). Learners make their own decisions about whether the knowledge, is 
important for them to obtain. Designing graphics for effective communication needs ^^^^^Z,. 
make predictions and evaluations from the information perceived (Reynolds & Baker, 1987). The graphics selected 
should be relevant to the message provided in the sense that they help visualize what needs to be learned. 
Additional cues or excessively realistic cues might become distractions to learners because irrelevant cues evoke 
c^mpetkive responses in opposition to desired learning. Although the effects of different levels of pictorial 
eS vary depending upon the levels of students prior knowledge, the pacing ot the instruction, and he type of 
educational objectives to be achieved (Joseph & Dwyer, 1984), it is important to present the design with a format 
that allows learners to capture the central concept of the message. 

Motive Matching for Providing Relev ance ...•„«,• 
To use graphics for relating learner's interests and understanding viewer s cultures and characteristics is 
important (HorL. 1991). Since graphics affect certain emotions, making predictions ^^.^TS^ 6 
from graphics is essential to using the graphics for achieving specific emotional purposes (Cate. 1993 Kovach, 
1988) The selection of graphic objects requires an understanding of the characteristics of the learners their 
Svfl^SSEJ «n5nt interest. It also requires creativity to link the instruction with the realistic world. 
Research has shown that people's responses obtained from pictures affect how information is perceived, stored, and 
Grieved (Langhinrichsen & Tucker. 1990; Lehmann & Koukkou. 1990). Relating graphics with things that 
interest viewers is expected to create memorable impressions and substantially enhance involvement in learning 
(O'Connell 1992). However, this approach should be employed with precaution. Since there exist differences in 
gender culture, language, or learning style among viewers, the effects, actions, or thoughts evoked by a graphic 
St vart among different groups of learners (ChanLin. 1993). Specific graphics that encourage the potential 
learnino motive of one group might not have the same impact on another. Consideration of individuals needs to 
be taken" when doing graphics design. One should carefully examine whether a specific culture or sex group is 
excluded from the design if the instruction is used to reach a larger group of viewers (Horton. 1991). 

Fmi'iiiritV fnr Providing Relevance , 

Tving new learning to what is familiar to learners can facilitate the comprehension and memory of new 
concepts lincc learning involves reconstruction and integration of new information with existing knowledge 
fGienc Brings & Wager. 1992). In computer lessons, graphics are used as cues to trigger the ideas from relating 
concepts and prior experience associated within the learning context. Since graphics arc used as a cue to help 
encode important information (Beck. 1987), they should be meaningful to the intended viewers (Cate 1993) If 
the graphic image is familiar and meaningful to learners, it promotes intuitive responding and facilitates encoding 
during the process of learning (Begg, 1983; Mealing & Yazdani, 1990). 

Abstractness and unfamiliarity arc two factors that tend to reduce curiosity and learning (Keller, 1983 Keller 
& Burkman, 1993). A graphical representation should take advantage of this principle by exploiting analogies 
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between an already know domain and a to-be-learned one. Analogy and metaphor can help learners relate new, 
unfamiliar, abstract knowledge to something that is concrete and familiar, and thereby capture its essence 
(Alesandrini, 1987; Cate, 1993). Computer graphics are often employed as a visual language to quickly 
communicate with certain groups of people who share a specific culture. If graphics are to be adopted by learners 
as a comprehension tool, they should be a visual representation that is easily grasped for referencing the 
information (Cate, 1993; Richardson, 1990). Effective use of metaphors and analogies is dependent on fully 
understanding the knowledge and the visual world from the point of view of intended learners (Cate, 1993). 

Confidence 

People's attitude toward success or failure has great influence on their learning performance. The important 
implications for providing motivation in the instruction is to make the learning materials assist a positive 
expectation for success and enhance the student's beliefs in their competence (Alesandrini, 1985; Keller <& 
Burkman, 1993). To increase confidence in success, instruction should be embedded with necessary elements to 
make students believe that the instructional materials are the tool to expected achievement. Therefore they will 
devote their effort in perusing the materials given. To encourage students to use pictures to decode or cue the 
important information for which the graphics are designed, carefully studying how students interact with those 
graphics, and designing graphics to fulfill students' learning requirements is essential. 

Learning Requirement for Providing Confidence 

Presenting learning requirements is critical to helping learners build a positive expectation for success (Keller, 
1983). Integrated into instruction, graphics can be employed as a primary source of information, an analogy to 
explain concepts, or a cue for focusing important information (Fleming, 1987; Hannafin & Peck, 1988). In order 
to fulfill the learning requirement and make graphics clear in the context represented, ihey should be consistent and 
correspond to textual messages, and presented simultaneously with corresponding text, so that students can inspect 
the graphics and the explanation together (Hannafin & Peck, 1988). In many cases, a single picture might 
contain important information to present a sequence of related concepts that are necessary for achieving intended 
instructional outcomes. Presenting a complex graphic should take into consideration differences in student's 
levels, abilities, and the types of instructional objectives that are to be accomplished. If a picture is too complex 
for consuming at one time, it should be dissected into smaller pieces (Fleming, 1987). If a picture contains too 
much information to present concepts, adding the details or complexity gradually is needed (Fleming, 1987). 

Positive Consequence for Providing Confidence 

Positive consequence is important in enhancing students' beliefs in their competence with the learning 
experience provided. A learning task that is too simple or too difficult might not encourage students to engage in 
the task provided (Keller, 1991; Malone, 1981). Graphics should be appropriate to task difficulty (Dwyer, 1988). 
Graphics containing too little information may not be adequate to challenge students and may cause them to 
become uninterested; graphics containing too much information may tend to overwhelm students and cause 
withdrawals from necessary interactions with the graphics which will enhance the learning experience. An 
appropriate level of learning difficulty is desired (Alesandrini, 1985). To maintain learners' intrinsic interests and 
quantity of performance, motivating feedback is suggested following responses. Although a lesson is more 
motivating when it challenges learners by providing graphical feedback on progress toward the learning goal, the 
use of graphics is not equally emphasized in all kinds of responses. The graphics should always reinforce and 
enhance positive learning rather than reward mistakes and failure (Alesandrini, 1985; Hannifin & Peck, 1988). 

Personal Control for Providing Confidence 

To encourage students to exert effort, the learning environment should provide the opportunity to allow 
students to accessing the important learning information at their own pace and at any time they want to 
(Alesandrini, 1985). Different individuals have different ability in learning from graphical materials. When 
learning under self paced conditions, sufficient learning time is essential for students to take full advantage of the 
information provided in the graphics. 

Students' confidence about learning specific content is also influenced by whether they feel they can easily 
navigate through computer learning materials and access information they need (Litchfield, 1993). Cognitive maps 
or other visual organizers are effective tools to help organize thoughts and clarify directions and lesson flow. 
They also provide substantial improvement in help retention. These cognitive maps not only help students to 
represent abstract or implicit information in more concrete form, depict the relationships among facts and concepts, 
but also help generate and elaborate ideas (McTighe & Lyman, 1991). Students gains substantial control over the ' 
materials when they know what they are learning ab.«ut and how they can proceed through the learning with the 
organization provided. 
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students minds (Friedman, 1993). Through experiencing and constructing, students are able to mentally visualize 
the objects and to form their knowledge framework (Morris & Hampson, 1983; Shepard, 1975) With the 

SlKfe^SS r^afce" ternally ' **" " 38 * ***** inferences in future knowl edge acquisition and 

Summary 

To accomplish motivational requirements, designed graphics should focus both on appeal in the sense of 
pleasing the eye and provision of the elements required to meet the learning tasks. Prior to development 
identifying what the instructional purposes are and how the mouvational strategies can be used in facilitating 
different mouvational purposes is essential. 6 
Although recent literature (e.g., Brown, 1992; O'Connell, 1992) has addressed the merits that computer 
graphics provide in enhancing interaction and exploration, these merits are supported only when the mouvational 
requirements are met. Motivation in learning computer or instructional materials, for learners, is the initial 
indicator influencing a learning process in a given task (Keller, 1988; Keller & Burkman, 1993; Litchfield 1993) 
How students approach, interact, and perceive computer or interactive learning materials is directly related to how ' 
they perceive the task and the situation given (Litchfield, 1993). 

Although in computer graphics design, a focus on the four motivational issues (attention, relevance 
confidence, and satisfaction) has not been emphasized in the literature, designing graphics to fulfill the ' 
requirements in gaining and maintaining students' attention, relating to their interests, building up their confidence 
in understanding, and satisfying their learning curiosity is an important factor in encouraging involvement in 
learning from computer-based materials. From a motivational aspect, to be interactive and motivating, computer 
graphics should be able to mimic the real world interaction, and encourage learners to think and to capture their 
imagination, and to help them move through their own creative ideas more easily. It becomes a challenging task 
for a graphics artist to generate graphics that are capable of facilitating learning motivation (Wileman 1993) 

Although graphic design requires proficient skills in the use of available software packages, to make graphics 
motivational, employing motivational design strategies is expected to move computer graphics toward a more 
effective approach. Carefully examining the motivation purpose of graphics used in relation to the instructional 
context provided, and the levels of learners in interpreting the graphic materials, is a critical process prior to the 
production of materials. Through the motivation framework provided, graphic designers are invited to view the 
designing task from a boarder perspective. With this approach, the different motivational factors among attention 
relevance, confidence, and satisfaction are focused upon. 
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Table 1 

Motivation Strategies for Graphic Design 



ARCS Model. KdUc HMD \ $Crat^>«9 for Graphics Desi&u 


Attention i 

! 

Perception Arousal 

What can I do to capture students 4 interest i 
in the computer learning materials? 

Inquiry Arousal 

How can I stimulate an attitude of inquiry? 
Variability 

How can I maintain their attention? 


What can be done in graphics to help attention getting and 

!i)iV;.'tain*/:^V 

• Use graphic: jjac are eye-catching to initiate learning 

iutcrest Rieber, 1989a). 

• Use visualization that is concrete rather than abstract to 
facilitate amending the concept (Dwyer, 1987). 

r Cue important elements in graphics to lead attention (Beck, 

mi) 

• Match graphic objects with learner's current interests 

(Alesanorini, 1985). 

• Use erapkc objects that stimulate curiosity (Alesandrini, 

1985). 

• Vary graphics provide for different interests (O'Connell, 

1992). 

• Employ various graphic formats and styles for sustaining 

attention (Haerte?, 1992). 


Relevance 

Goal Orientation 

How can I relate the instruction to the 
learner's goals? 

Motive Matching 

How and when can I link my instruction 
to the learning styles and personal interests 
of the learners? 

Familiarity 

How can I tie the instruction to the 
learner's experiences? 


What can be done in graphics to establish students' perceived 
relevance in learning from the instruction? 

• Use graphics relevant to the purpose that the instruction 

intends to achieve (Reynold & Baker, 1987). 

• Match design objects with contemporary events or learner's 
current interest (Alesandrini, 1985). 

• Relate design with students' characteristics (Horton, 1991). 

• Make graphics individualized but not exclude specific group 
of learners (Horton, 1991). 

• Use meaningful representation to convey information 

(Begg, 1983). 

• Use analogy and metaphor to convey unfamiliar and 

abstract concepts (Cate, 1993). 
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Confidence 

Learning Requirements 

How can I assist in building a positive 

expectation for success? 

Positive Consequences 

How will the learning experience support 

or enhance the students' beliefs in their 

competence? 

Personal Control 

How will the learners clearly know their 
success is based on their effort and 
abilities? 



What can be done in graphics to establish students' 
confidence in learning the computer program? 

• Correspond graphics with text (Hannafin & Peck, 1988). 

• Dissect complex graphics into digestible pieces (Fleming, 

1987). 

• Allow appropriate challenge level and full interaction with 
graphics (Alesandrini, 1985). 

• Use graphics to reinforce positive responses rather than 
negative responses (Hannafin & Peck, 1988). 

• Provide a sense of control and choices for accessing 
graphical information (Alesandrini, 1985), 

• Provide a tool for organizing information through graphics 
(McTighe & Lyman, 1991). 



Satisfaction 

Intrinsic R einforcement 

How can 1 encourage and support their 

intrinsic enjoyment of the learning 

experience? 

Extrinsic Rewards 

What will provide rewarding consequences 
to the learners' success? 



Equity 

How can I build learner perceptions of fair 
treatment? 



What should graphics do to establish satisfaction during the 
time of learning through graphics? 

• Use graphics to encourage exploratory thinking and 
transforming skills (O'Connell, 1992). 

• Use well produced graphics for intrinsic reinforcement 
(Levie, 1987). 

• Provide more interesting graphics when the tasks require 
more processing effort (Peeck, 1987). 

• Provide opportunity for practicing and experiencing success 
from graphics (Fitzgerald, 1989). 

• Help visualize the real environment through graphics 
(Brown, 1992). 

• Encourage students to apply and relate what they study 
from the graphics to real world situations (Dwyer, 1988). 



* The subcategory used here is from the older version of ARCS Model, because "personal control" is more related 
to the graphics strategy described. 
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Abstract: The main goal of schools is to help students learn. In a 
modem society rich in information, knowledge and skills, this requires 
schools to provide learners with the knowledge and skills necessary for 
full participation in the dynamic changes of societal evolution. 
However, while students spend almost one-third of their awake-hours 
in school, how much actual learning can be applied to their interactions 
outside of school? Not understanding new learning material or one s 
inability to transfer that learning to other domains has not only 
educational costs but also personal costs. The educational institution 
loses time and money through lack of adequate student preparation and 
the subsequent need for re-teaching. Individuals may develop learned 
helplessness and fear of school through constant exposure to 
"unintelligible" teaching and from the rapidly advancing norms of 
society. 

The Duroose of this study was to assess the effectiveness of techniques that enable students 
to transfer learning across knowledge domains, analogical reasoning. Analogical rea sorung as a 
pr^osed intervention appears to be conceptually consistent with recent ideas and developments 

111 °SSSraont concept relative to helping students understand new learning is by relating it 
to Drior knowledge. However, not every student possesses prior school learning to relate new 
XSSg elements to. To overcome this obstacle analogical reasoning [techniques can 
provide bridges to other analogous prior knowledge outside of the content area (Curtis & 
Reigluth, 1984). 

Review of Literature 

Prior Knowledge 

From the perspective of cognitive psychologists, learning is a process of relating new 
information to previously learned information. It is similar to Piaget s assertion that 
^SSSSSkM al ways involves assimilation (Piaget, 1970). nformation processing 
theorists believe that "learning is most likely to occur when individuals can rdatsa new 
experience to the information they have acquired from earlier experiences (Omrod, . 990 
Ausubel (1960) proposed that learning supported by poor knowledge can serve as an advance 

0r8 Craik and Lockhart (1972) proposed that what gets encoded into memory depends upon the 
level or depth of processing of the presented information as it is perceived and encoded into 
memory during the learning process. As the level of processing deepens, to more 
information will be recalled because more meaning will be assigned to it (Ormrod, 1V9U). 

However, people cannot always perfectly retrieve encoded inforrr ^fp^ d J. m ^' Tf 
"attributed to deficient retrieval strategies rather than simply forgetting (Ormrod, 1990). It 
prior knowledge can be provided, learners will not have retrieval strategy problems. 
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presented before "Presenting stimuli with distinctive features" (Gagne, Wagner & Rohas, 
1981, p. 182). All of these assertions support that accessing prior knowledge before 
instruction is very effective for learning. 

Analogical Reasoning 

Analogical reasoning is to relate what is to be learned to something with which the learner 
is already familiar (Gentner, 1983). This relation compares the new idea to an highly similar 
one that is known (Ortony, 1979). Analogical reasoning can be distinguished in two ways. The 
first one is classical analogies which involve four elemental comparisons in two main parts: 



The second one is problem analogies that involve only two elemental comparisons in two main 
parts as exhibited in a learner recognizing the relational similarity between the solar system and 
the Bohr atom model. Goswami (1991) further identifies Piaget's structural theory and 
Sternberg's information processing theory as classical analogies and Gentner's structure 
mapping theory as problem analogies. 

The effective aspect of analogies also depend upon the learner understanding a set of base 
knowledge (Gentner & Gentner, 1983) and noticeable relational commonalities between base 
and target domains (Getener, 1988, 1989). (Gentner & Gentner, 1983). Gentner also argues 
that the key to successful use of analogies is to notice relational commonalities between target 
and base domain, for example in the orbital nature of planets in the solar system and of 
electrons in a Bohr atom. 

Novice and Expert 

The difference between experts and novices in identifying problems is that "expercs have a 
more complete and better organized knowledge base for the problems they solve (Cochran, 
1988; Ormrod, 1990). Experts not only do know better which concepts should be associated 
with which, they also know the particular relationships that different concepts have with one 
another. Novices are often uncertain about how the different knowledges fit together and are 
more often wrong about how the concepts are actually interrelated, focussing on superficial 
physical characteristics rather than on functional characteristics. 



Using analogies to understand a new domain is widely accepted as playing a role in 
learning behavior. The application domains of analogies have been a diverse as science 
education (Gentner, 1977; 1983) and reading (Goswami, 1986). Clement et al. studied the 
consequence of analogical reasoning in the learning of programming languages (Clement, 
Kurland, Mawby & Pea, 1986). However, little attention has been focussed recently on the use 
of analogies to assist novices in learning computer languages. 

Due to the novice level of the subjects in this research, no explicitly parallel schema can be 
assumed to exist in their knowledge base. Therefore, a teaching strategy using analogical 
reasoning was modeled according to an environment the students were likely to have 
experienced elsewhere. To that end, the process of cooking involved in the preparation, 
cooking and presentation of a meal is assumed to be common enough to be considered base 
knowledge for all participants. This cooking analogy is accessed in the lesson presented in this 
research. 

The Research Project 

This research was designed to inspect the effectiveness of analogies in improving the 



bird : feathers -> dog : hair 



Statement of the Problem 
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relative to BASIC computer programming concepts via computer-assisted-instruction (CAI) for 
both male ana female learners. 

Subjects 

Novice learners in this study are defined as persons who have little or no previous 
experience with computers and lack specific knowledge of computer programming (Mayer, 
1981). 

24 students from an undergraduate Instructional Technology course, EDIT 2318, 
"Cbmputers and Technology" at Texas Tech University volunteered to participate in this CAI 
lesson in return for "extra credit" grading. 

Materials and Equipment and Facilities 

Due to its inherent graphic capabilities, the ease with which a developer may incorporate 
interactive "stacks" (programs), and one of the researcher's experience with its development 
language, Claris "HyperCard™" for Apple Macintosh® computers was chosen as the media 
for delivery of the lesson. Lesson content and performance objectives were assessed and 
verified by content area experts; Dr. Bob Price, PhD., Mr. Roger Von Holzen and Mr. Donald 
J. Winiecki of Texas Tech University. 

The site chosen for delivery of the lesson was the Educational Computing Center (ECC) 
lab, rooms AD325 & AD326 on the Texas Tech University Campus. Students enrolled in the 
EDIT 2318 course are required to use this laboratory for many of their class assignments and 
are familiar with its location and environment. 

A total of five Apple Macintosh computers (four model SE and one model SE/30) were 
available for use during the lessons. Each of these computers is comparable in screen 
resolution, processing speed and operating characteristics relative to the needs of the presented 
CAI lesson. 

Lesson times accommodated student and teacher schedules and varied from 8:00am to 
4:00pm on a Thursday through Tuesday schedule. 

The ECC lab was open for normal student use during some of the lesson sessions and 
activity during the lesson sessions ranged from low to very busy with a corresponding range of 
noise and potential distractions. Any effects resulting from these environmental changes is 
unknown. Although during lesson observations, some subjects appeared to be distracted by 
various activities in the room. 

Instrument 

A researcher-made test of 20 multiple-choice questions was developed. The purpose of this 
achievement test was to measure subjects understanding and mastery of BASIC programming 
concepts presented in the CAI lesson. Face validity of this test was was assessed and verified 
by content area experts; Dr. Bob Price, PhD., Mr. Roger Von Holzen and Mr. Donald J. 
Winiecki of Texas Tech University. 

Post research analysis using the KR-20 instrument indicated an inter-item consistency of 
.71 among the study's 24 participants. 

Procedure 

Two weeks before taking the CAI lesson, the subjects' regular instructor explained to them 
the nature of the research project and administered a consent form for them to sign. In addition 
to the consent form, each potential participant completed a pre-intervention questionnaire 
(appendix A) containing three subtests; a Computer Attitude Rating Scale, Computer Anxiety 
Rating Scale and a Self-Efficacy Scale for computer use. Students participating in this research 
were awarded "extra credit" for their cooperation. 
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Subjects were randomly assigned to one of two treatment groups: 

Experimental Group 

The experimental group received a CAI lesson presenting primary BASIC 
programming concepts with consistent use of analogies, metaphor with the analogies 
and metaphor embedded in the instruction. 

Control Group 

The control group received a CAI lesson presenting the identical primary BASIC 
programming concepts using onscreen text only. The control group was not exposed to 
any use of analogies or metaphors in the presentation of the CAI lesson. 
Immediately following the intervention each subject was asked to complete a posttest and a 
post-questionnaire (identical to the pre-intervention instrument). 

Two weeks following the intervention, subjects returned to complete a delayed test 
(identical to the post-test) to measure student's retention. 

Results 

Mean scores on the immediate post-intervention test were: control=76.15, 
experimental=74.09. Mean scores on the delayed test were: control=70.49, 
experimental=71.07. Analysis of variance revealed that there was no statistical differences 
between groups at the .05 level. Results are reported in table 1. 

The results show that the control group had the highest score in both immediate and 
delayed post-test 

Item analysis of the post-test revealed that its item difficulty ranged from 25.49% to 100% 
and point item biserial discrimination achieved values of +0.75 to +0.494. 

Average attitude toward computers increased ,29, anxiety decreased .38 and self efficacy 
increased .25 within the control group. The experimental group's attitude toward computers 
increased .5, anxiety decreased .56 and self-efficacy increased .61. Results are reported in 
table 2. 

Inter-item reliability of the questionnaire is presented in table 3. 

Analysis of Results 

From an analysis of variance, we found that the mean of post-test, delayed test, post-test 
anxiety and post-test self efficacy did not undergo significant change. However, significance 
was realized in post-intervention attitude with a covariate of pre-intervention attitude (P=3.24, 
P-0.031). 

Interpretation of Results 

The effects of activating a well known analogy as a schema during CAI was investigated. 
Apparently, the experimental treatment was too weak to have an effect on dependent variables. 
In brief, the assumption of using analogies to increase the attitude and self-efficacy and to 
decrease the anxiety toward computers and computer programming was not supported between 
groups. 

Conclusions 

A basic premise of this study is that learning does not occur in a vacuum. Learners must 
have appropriate schema to understand incoming information. In other words, learning is to 
have successful retrieval of appropriate schema from the long-term memory to give meaning 
and structure to the new information accepted into working memory and encoded into long 
term memory (Norman, 1988). 

However, for the novice who may have no appropriate knowledge base, relating new 
knowledge to highly similar knowledge that is external to the learning domain is of interest 
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(Reigeluth, 1983). The subjects in this research generally had little background knowledge of 
computer programming languages. For that reason, some way was needed to make the schema 
more concrete and understandable for them. This was attempted using analogy and metaphor. 

For the above reason, we chose to present a lesson on the systematic nature of computer 
programming through metaphors and analogies representing a process that our students were 
likely to have had previous experience with. The metaphor and analogies around which this 
lesson is constructed are created by posing the computer itself as a humanoid robot engaged in 
activities normally experienced during the preparation, cooking and presentation of a meal 
(preparation, input, output in the programming process, appendix B). We thought the 
representation of the computer as "a thing to direct orders to" (eg: a robot) and the use ot 
cooking procedures to model the programming process would provide an environment within 
which anxiety may be reduced and instruction presented in a f?miliar and more easily encoded 
format. Additionally, we expected test performance to incre *e, attitude toward computers to 
increase, self-efficacy toward computers to increase and anxiety caused by computers to 
decrease. 

Disadvantages of this study include: 

1. This CAI program contains no practice session, subjects were not given the opportunity 
to act upon new information. # 

2. Although novices to computer programming, subjects had widely varying expenences 
with computers in general. . . 

3. Subjects attitude toward the metaphor and analogy used in this lesson may have had 
some effect on results. ... 

4. Subjects were not prompted to use any presented analogies during post-intervention 

tests 

5. The intervention was only administered once for a period of approximately 20 minutes. 
Additional sessions or longer duration may have some additional effects. 
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Appendix A. Computer Anxiety, Attitude, and Self-Efficacy Survey: 

Adapted from Harrison & Rainer (1992). 

EDIT Password: 

For the questions (1-12) below, please circle the initials (eg: SD, N, SA...) representing how 
strongly you agree or disagree with each statement. 

Use the following scale to rate your responses: 

SD = Strongly Disagree A = Agree 

D = Disagree SA = Strongly Agree 

N = No Opinion 

1 . Computers can eliminate a lot of tedious work for people. SD D N A SA 

2. Computers make me uncomfortable because I don't understand 

them. SD D N A oA 

3. Computers intimidate me because they seem so complex. SD D N A SA 

4. Computers are difficult to understand and frustrating to work 

with. i>D D IN A oA 

5. I do not think I would be able to leam a computer programming 

language. i>D UNA oA 

6. I dislike working with machines that are smarter than I am. SD D M A SA 

7. The challenge of learning about computers is exciting. SD D N A SA 

8. I am confident that I can learn computer skills. SD D N A SA 

9. 1 feel confident working on a personal computer (miCTOComputer). SD D N A SA 

10.1 feel confident understanding the three stages of data processing SD D N A SA 
(input, processing, output). 

11. 1 feel confident using the computer to organize information. SD D N A SA 

12.1 feel confident writing simple programs for the computer. SD D N A SA 
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Appendix B. Hard copy output of CAI lesson screens. 



BflSICflM 



iiii 



Computers donl understand English, 
they only understand s language 

called "machine code" that we can't 
read. 

Instead, there are many "computer 
languages 1 ' like; BASIC, Pascal, 
FORTH, C and others. 

We translate English instructions into 
a computer language and the 
computer can then translate It Into 
machine code. 

In module 7 of EDIT 2318, we'll send 
instructions to the computer using 
BASIC. 



Typical Experimental Group Screen 




Computers donl understand English) they only 
understand a language called "machine code" that we 
canl read. Instead, there are many "computer languages" 
like; BASIC, Pascal, FORTH, C and others. 

We translate English instructions into a computer 
language and the computer can then translate it into 
machine code. 

in module 7 of EDIT 2318, we ll send Instructions to the 
computer using BASIC. 
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Typical Control Group Screen 
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Table !• Posttest and delayed posttest. (n s 24) 





n 
11 


Pnctfp^t Mean 


Delav test Mean 


Adj. Mean 












Control 


13 


76.15 


73.46 


70.49 


Experimental 


il 


74.09 


72.27 


71.07 












F ratio 


0.55 


1.118 


2.156 


P value 




0.65 


0.352 


0.107 



Table 2. Questionnaire Survey, (n = 24) 







Pre- 






Post 




Differences 




Att 


Anx 


Eff 


Att 


Anx 


Eff 


Att 


Anx 


Eff 






















Control 


3.67 


1.98 


3.52 


3.96 


1.6 


3.77 


0.29 


-0.38 


0.25 


Exp. 


3.18 


2.18 


3.16 


3.68 


1.82 


3.77 


0.5 


-0.36 


0.61 























Table 3. Inter-item Reliability (Alpha) on Questionnaire, (n = 24) 





Pre- 


Post 








Attitude 


0.72 


0.70 


Anxiety 


0.73 


0.72 


Self Efficac 


0.78 


0.74 
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Using Computer-Based Microworlds with Children with Pervasive 
Developmental Disorders: An Informal Case Study 



LLOYD P. RIEBER 
The University of Georgia 
Department of Instructional Technology 
620Aderhold Hall 
Athens, Georgia 30602-7144 USA 
Email: LRIEBER@MOE.COE.UGA.EDU 



Abstract: This paper presents an informal case study of an I l-yearoldchild*sinteraction 
with several computer-based applications and microworlds. The child, diagnosed as 
having Pervasive Developmental Disorder resulting in multiple learning disabilities, has 
interacted extensively with a Macintosh computer since September, 1991. He has used 
several commercially available graphics packages as well as a series of specially designed 
software programs in the areas of language, mathematics, symbol manipulation, and 
general problem-solving. The child's case history is discussed both in general and in 
regard to his computer experiences. Examples of the instructional design and develop- 
ment of the customized computer microworlds, using rapid prototyping procedures, are 
also discussed. 

This paper is about a child named Thomas. Thomas has significant developmental and behavioral disabilities 
that have presented him with unique challenges to learning, social interaction, and social acceptance. Consequently, 
he has great difficulty with the simplest of learning tasks. However, he is slowly finding an important ally in the 
computer. This paper tells the simple story of how Thomas and the computer have begun to work together as partners 
in cognition — a story that continues to evolve. 

This paper is written from a special point of view because I am Thomas* father. For this reason it is written from 
my background in instructional technology, not special education. Likewise, there is no pretense in trying to present 
this story in any objective way. My work with Thomas stems from an admittedly selfish base. However, it is hoped 
that many will find some value and relevance in this story despite the inherent bias. At the very least, it is hoped 
that this paper may cause some to reflect more deeply on how to celebrate the diversity in all people. 

Thomas* story will be told through the computer software he uses. Some of the software (i.e. MacPaint, 
HyperCard, KidPix, The Playroom) have been commercially produced, whereas I custom built other software for 
him. It would not be accurate to say, though, that / designed the software for Thomas. In reality, he should be 
considered as a "co-designer** because the most successful software I* ve developed for him have been based on his 
unique abilities and interests. Either the software reaches Thomas on his level or it simply does not reach him. As 
a result, Thomas has taught me a great deal about software design. 

The purpose of thispaper is to describe Thomas* progress and obstacles to understanding critical aspects of both 
general symbol manipulation and general computer operation. In Thomas* case, each has been wedded to the other. 

Background 

At the time this project began, Thomas was 9 years old. He had exhibited several severe learning disabilities 
from about the time he was one year old. In addition, he exhibits both severe developmental delays resulting from 
language disorders (e.g. form, content, and function uses of language) and speech disorders (e.g. articulation), 
though it would be misleading to suggest that the two areas are independent of each other. Most developmental 
psychologists agree that language processes play a central role in intellectual development, though the issue of 
whether language determines intelligence or visa versa is still widely debated. 

Thomas has difficulty with even the simplest intellectual skills, such as making simple discriminations. His 
reading skills are limited to recognition of only several letters and a few sight words. His mathematical skills are 
confined to simple counting from 1 to about 5, though he docs so apparently from rote. His construction of number 
theory usually seems limited to 0, 1 , 2, and many. Thomas' writing ability is limited to crude representations of only 
a few letters and numbers. The only word he is able to write with accuracy is his nickname — 'Tom.'* Thomas is 
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severely deficient in some of the most fundamental cognitive abilities that we all take for granted — simple language, 
simple mathematics, and basic social skills. Therefore, his view of the world is likewise limited. His "knowledge" 
seems completely tied to specific situations and tasks with general transfer severely limited. 

Thomas also exhibits behavioral disorders. He typically goes through periods of intense frustration and 
aggressive behavior for days at a time. Occasionally, he will experience a severe episode where he can no longer 
control himself and he will seem filled with rage. These are obviously very difficult periods for our family (we also 
have a 14-year old daughter). We are beginning to see a correlation between these extreme episodes and times of 
general stress in the family, such as very busy times, inconvenient travel schedules, and moving. In contrast, Thomas 
appears to most people as a very sweet child otherwise, though his responses and behaviors to simple social 
interactions often appear odd and confusing to people, especially strangers. He also has a very good sense of humor. 
Though it may be an oversimplification and too speculative at this point, we have noticed that Thomas' most 
disruptive behavior is often tied to times when he does not have control over a situation, whether due to an inability 
to comprehend what is going on or due to physical limitations (such as going on long car trips). 

From a pure Piagetian perspective, Thomas' developmental level appears to be somewhere in the preoperational 
range. For example, Thomas is just beginning to develop temporal abilities that extend beyond to something 
happening the next morning or the next day. Thomas shows no sign of being able to conserve, though it is difficult 
to test Thomas because of his language comprehension disabilities. Thomas is almost entirely ego centered. He 
seems to view the world only in terms of how it affects him, a typical characteristic of this stage. 

After receiving extensive clinical evaluations, he was diagnosed as having pervasive developmental disorder 
(PDD). In addition, he has been diagnosed as having attention deficit-hyperactivity disorder (ADHD). PDD is a 
general description given to individuals exhibiting a wide range of developmental disabilities (Brown & Reynolds, 
1 986) and is defined as children "in which many basic areas of psychological development are affected at the same 
time and to a severe degree" (American Psychiatric Association, p. 34). This term is most often used to classify 
individuals whose learning disabilities do not easily fit other existing taxonomies, though PDD is usually grouped 
together with autism (but even on this point there is wide disagreement) (Bullock, 1992). People with PDD have 
learning problems that are related to a multiple of factors. The term PDD is used to describe a disorder that is simply 
not yet well understood. As is typical, a label often precedes understanding, but with it comes the risk that some 
will substitute the label for understanding. 

Despite these extensive developmental and behavioral challenges, Thomas has many talents and aptitudes. He 
is musically inclined — he shows a good ear for melodies and lyrics and is particularly fond of musicals. He has 
memorized melodies and lyrics for entire songs (even though he has been unable to memorize individual letters of 
the alphabet in this way). He has very good visual skills. Drawing is oneofhis favorite pasttimes. Hedemonstre is 
a natural ability to coordinate colors (e.g. he chooses his clothes carefully with matching or complementary colors). 
He is also becoming very adept at operating a Macintosh computer, an ability on which the rest of this paper is based. 

In September, 1991, various computer activities were introduced into Thomas's home school curriculum .' 
Thomas has been given on average approximately two hours of structured computer time each week in a university 
setting combined with casual access to a computer at home, resulting in approximately 10-15 hours of computer 
interaction per week on a continual basis. Thomas has extensively explored the graphics capabilities of several 
simple packages such as MacPaint* KidPix, and HyperCard. In addition, several computer-based activities were 
specifically designed to give Thomas simple and structured experiences in language and mathematics. These 
customized materials, designed and developed using rapid prototyping methods, have been in the areas of simple 
mathematics, letter recognition, shape recognition, general symbol manipulation (e.g. meaning of arrows), and left/ 
right differentiation. Other goals of the materials were to improve Thomas's general cognitive skills (e.g. increasing 
his ability to selectively attend to components of a task), to establish a meaningful context for interacting with 
language and mathematics, to increase his self-esteem and self-efficacy, and to increase his ability to work 
independently. His use of the graphics applications and many of the customized software programs conforms to the 
use of computer microworlds as defined by the constructivist paradigm to learning (Rieber, 1 992), though tempered 
with extensive use of game-like features (see Rieber, 1994a, for an extended discussion of blending the 
characteristics of microworlds, simulations, and games). 

The computer is becoming an ever increasing ally in Thomas's intellectual repertoire. He has entered into what 
some call "cognitive partnerships" with the computer (Salomon, Perkins, & Globerson, 1991). His skills and 
abilities are closely aligned with the capabilities of the computer. Just as a noimally functioning person still relies 
on the media of pencil and paper to supplement memory limitations, Thomas has begun to rely on the computer for 
many of his intellectual achievements. 

The First Encounters 

Thomas' first computer experience was with Kids on Keys, an older letter recognition package on an IBM- 
compatible computer that uses animation in a drill and practice format. The user must find and press the letter on 
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the keyboard corresponding to an animated letter slowly "falling** on the computer screen. If the user is successful, 
the computer graphically "zaps" the letter. Thomas enjoyed the game despite a complex interface that frequently 
interrupted his play. For example, after correctly "zapping** a certain number of letters, the computer prompts the 
user to proceed to the next level. Early on, this always required adult intervention, although interestingly, Thomas 
quickly learned to imitate the key press procedure. Still, the package always required some adult intervention at 
some point. The package also did not contain any audio elements, so Thomas was learning only to discriminate on 
the letter shape, not the letter name. Given these limitations, I decided to custom design a similar package for 
Thomas. 

The software I designed, called Letter Match, also had 'letter recognition objectives similar to Kids on Keys. 
However, the other intent also was to familiarize Thomas with software using a mouse interlace, specifically the 
concept/procedure of "aim and click.** This second goal turned out to be the real value of Letter Match because the 
primitive mouse skills that Thomas achieved with Letter Match became the building blocks for interacting with all 
of the other software described in the rest of the pap^r. Thomas is now very versatile with the mouse interface and 
uses the mouse as an "extension of self* like any other Macintosh or Windows user. 

Letter Match begins with a digitized picture of myself followed by the audio message "Hi, Tom ! Point and click 
on my picture to begin.** The audio message is repeated after about 20 seconds. When successful, the audio message 
"Thank ycu!** is given and the software proceeds to the letter recognition task. All other input is ignored by the 
computer. The letter recognition task uses only the keyboard, just like Kids on Keys. After three successful letter 
matches, the software branches back to the mouse activity using my picture. At first, I had to help Thomas with the 
point and click task by guiding his hand and finger on the mouse. By the end of the first session, he was able to 
perform the task alone with only limited intervention. 

The software then repeats this procedure until the someone quits the software using a pulldown menu option, 
a skill he has now completely mastered although it is not clear whether he has developed a rudimentary sight word 
vocabulary to include "Quit** or simply uses the spatial arrangement (i.e. last word on list) to make the correct choice. 
Fortunately, "Quit** was the only choice in the pulldown menu in Letter Match. 

Thomas still chooses Letter Match on a periodic basis, though I think it is just for the purpose of reliving an 
exciting discovery which became a starting point for his other computer adventures. 

Learning to Recognize, Understand, and Manipulate Abstract Symbols 

After a few sessions m\hLetterMatch f I introduced Thomas to MacPaint, one of the simplestgraphics packages 
on the Macintosh, with the hope that his graphical interest and ability would intrinsically motivate him to turn the 
computer into a graphical "construction set.** His "aim and click** skills were sufficient to allow him to begin 
drawing. Early on, even though his fine motor skills with the mouse were quite poor, this did not seem to interfere. 
He seemed more interested in the "phenomena** of making lines and scrawls on the computer screen than drawing 
any recognizable picture. I also introduced him to MacWrite, a simple word processor, but he showed absolutely 
no interest with it. Even now, for whatever reason, when he is working on the computer he continues to avoid most 
uses of symbolic language, such as letters, words, and sentences. 

For the first few months, his computer experiences were limited to MacPaint and Letter Match. He chose to 
split his time fairly evenly between the two. When working with MacPaint, Thomas relied on me to first choose 
the "large paint brush** tool from the program*s tool palette. At first, he was completely unaware of any other 
function. Interestingly, he quickly identified the interesting and unexpected goal of filling up the entire screen as 
though he were painting a wall. When he achieved this for the first time, I changed the pen pattern from black to 
a light shade of gray. He proceeded to paint over the entire screen again using this new pen shade. The goal of 
painting the entire screen, with varying pen patterns and shapes, became his primary "mission** when working with 
MacPaint. 

When working with MacPaint, he never showed any overt understanding of how a screen symbol, such as the 
"eraser,** would change the function of the mouse. However, anytime I intervened to change tools for Thomas, I 
was always careful to describe what I was doing. For example, after filling up the screen, I would ask him if he wanted 
to erase it. If he said yes, 1 would say, "OK, 1*11 click on the eraser** while demonstrating it for him. He would then 
take over the mouse and begin erasing. Obviously, Thomas was very dependent on mc in his use o\ MacPaint at 
this time. His standard routine was to draw with one tool and then ask for help to change to another tool. 

Thomas* first real breakthrough occurred about six months later when he made the discovery of how to choose 
a d.-ffercnt graphical tool for himself. It is easy to take for granted the conceptual leap that is necessary in 
understanding how one input device such as the mouse can assume multiple functions. This conceptual 
understanding is necessary for successful interaction with even the simplest software programs. While it might take 
a normally functioning 10 year old child only minutes to form this concept, it took Thomas six months. 

This breakthrough began Thomas* adventures with the computer that closely conform to the constructivist idea 
of a microworld. He appropriated the technology to fit his needs and expectations while being engaged in authentic 
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Figure I. Snapshot of the screen at the start of Sour Apple, 



experiences in a variety of contexts and domains, including mathematics, spatial manipulation, and language. 
MacPaint was replaced with the graphical environment of HyperCard, due to some technical problems I 
encountered with the latest Macintosh system software. Fortunately, the graphics of HyperCard are almost identical 
to MacPaint, though HyperCard uses different types of too! and pattern palettes. Interestingly, Thomas has learned 
how to access these palettes by "tearing them off* of the menu bar and the placing them just where he wants them. 



Based on Thomas' success with MacPaint, I introduced him to a very creative and imaginative graphics package 
for children called KidPix. This, by far, is Thomas' most favorite computer activity. Like MacPaint, Thomas uses 
KidPix to create graphics, though with color and with tools that also offer interesting sound effects (such as the 
sounds of a pencil "scratching" against paper and a very "liquidy" sounding paint bucket.) 

There are two activities that dominate Thomas' time in KidPix. The first activity, called "hidden picture," is 
built into KidPix. "Hidden picture" is a special erase feature where one uses the eraser to uncover a picture hidden 
"below." Thomas has spent so much time with this feature that he is able to identify the picture soon after the first 
details emerge. The second activity is an original creation by Thomas. Thomas draws random lines all over the 
screen, though most are vertical. He then quickly changes to the fill bucket tool. Then carefully and methodically, 
he fills in each small space with paint until the entire screen is virtually covered, similar to a variation of his "paint 
the wall" goal with MacPaint. Interestingly, he has extraordinary fine motor control and can "aim" the fill bucket 
to pour paint into spaces that are only a few pixels wide. Thomas seems to find this activity exceedingly relaxing. 



All of Thomas* experiences with KidPix, MacPaint, and HyperCard were worthwhile. He was engaged in 
imaginative play using a variety of tools and he was continually learning new tools as well as new ways to use old 
tools. On the other hand, his experiences were quite haphazard and there was little opportunity for him to explore 
many fundamental concepts of language and mathematics. There were also some basic skills that he was having 
much difficulty with, such as the concept of direction and discriminating between left and right. In response, 1 began 
to develop some activities for Thomas in which he would be less likely to become distracted by a myriad of functions 
and tools. The design methodology 1 used was really a hybrid of all my design knowledge, though the methodology 
of rapid prototyping comes closest to describing it (see Tripp & Bichelmeyer, 1990, and Rieber, 1994b, for more 
detailed discussions of rapid prototyping in instructional design). Consequently, the goals of the activities often 
emerged based on how he reacted to them. The computer authoring tools at my disposal allowed me to quickly 
develop a prototype to try out with Thomas in order to discover if the idea was worth pursuing or abandoning. All 
of these activities were created in a very short period of time, usually less than two days, and fine tuned based on 
Thomas* success, interests, and frustrations. The software will be briefly described in the next sections. 

Sour Apple. Sour Apple is a mathematics game to help Thomas develop basic number theory. The goal of the 
game is simply to eat all of the apples. As shown in Figure 1 , the game is played by clicking on the numbers one, 
two, or three. When one of the numbers is chosen, the number name is spoken by the computer and the appropriate 
number of apples are "eaten" (i.e. each shrinks to a vanishing point at their center). An important feature occurs 



Adventures with KidPix 



Custom Designed Software: Toward Microworlds 
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towards the end of the game. If one chooses "3" or "2" when fewer apples are actually left, the computer says matter 
of factly "But there are only two [or one] apples left!" while displaying this text on the screen. When all the apples 
have been "eaten," the computer says "You ate all the apples!" again with the text displayed on the screen. The 
decision to display the corresponding text was based on trying to reinforce the concept that written symbols (i.e. 
words) carry meaning. 

Though Thomas chooses to play Sour Apple on a regular basis, he rarely plays it for extended periods of time. 
He will usually play one or two rounds and then quit to choose something else. His typical strategy is to choose to 
eat only one apple at a time. He only occasionally chooses to eat two apples and he rarely chooses three apples. The 
original inspiration for Sour Apple came from a classic math game called Poison Apple which I had transferred to 
the computer about ten years ago (though I preferred to call it "sour apple"), I played the game with Thomas once 
and although it was filled with prompts and messages that he found confusing and annoying, he continually asked 
to play the game again. Consequently, I designed this version to meet his request and expectations. 

Shape Art. This is one of Thomas' favorite games. There are many purposes to Shape Art, including spatial 
manipulation, language based on sight word vocabulary, and simple problem solving which stresses part to whole 
reasoning. Shape Arts builds on Thomas' graphical interests and abilities by allowing him to draw pictures. 
However, instead of drawing with a pencil or brush one draws pictures with the three basic shapes of circles, squares, 
and triangles. As shown in Figure 2, there is a "well" of a limited supply of these three basic shapes on the bottom 
of the screen. One must click, hold, and drag one of the shapes to any location on the screen, a skill which Thomas, 
surprisingly, had little difficulty mastering. Thomas has the choice of drawing with the figures on a blank screen 
or choosing a "picture outline" corresponding to the buttons on the right side of the screen. When a button is pressed, 
the computer pronounces the word on the button, such as "train," and an outline of a train appears. The pictures can 
all be made by combinations of the three basic shapes. Thomas enjoys moving the appropriate shapes into their 
respective places. In addition, the parts to whole problem has a puzzle-like quality that Thomas seems to enjoy. One 
feature oi [Shape Art that Thomas particularly enjoys is the "Clean up" button. Its function is to put all of the shapes 
back to their original positions (using animation) and also erasing the screen. 

One intended goal of Shape Art is to provide Thomas with an interesting environment in which he can explore 
simple language, again in the hope that he will associate printed words with meaning. A future goal is to find 
activities in which Thomas can build asight word vocabulary through meaningful interaction with single words. Part 
of this motivation is due to Thomas' inability to make any sense of a phonics approach to language. The use of a 
heavy phonics approach even with normally functioning children is continually debated among language educators. 
In Thomas' case, phonics has become a psychological obstacle which I speculate is based on his frequent failure in 
the public schools. Thomas seems to "turn off or "tune out" any attempt to teach phonics. 

Treasure Hunt. The purpose of this activity, as illustrated in Figure 3, was to provide Thomas with a "left vs. 
right" microworld. The microworld is presented in a game-like context based on the fantasy of pirate treasure. This 
context was chosen based on Thomas' fondness at the time for the movie "The Goonies." When the graphic of either 
hand is clicked, the computer says cither "left" or "right" and the boat moves one step toward that direction. The 
most obvious goal is to maneuver the boat toward the treasure at the bottom while also avoiding the pirates (as 
indicated by the "cross bones" Hag) or sea monster. The graphics of the treasure* pirate flag, and sea monster are 
placed at random locations at the start of each round of the game. Of course, another goal might be to aim the boat 
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Figure 3. Snapshot of the screen during Treasure Hunt. 



toward the pirate or monster. In either case, the value of the interaction remains the same. This activity was a short- 
lived favorite with Thomas. Though he interacted with he often at first, he rarely chooses it anymore. His interest 
in the fantasy context has apparently dwindled. Of course, he hasn't watch the movie lately either, so perhaps that 
might trigger renewed interest in this game. 

Tom 's Train. The inspiration for this activity was setting up a toy train under the tree one Christmas. Thomas 
enjoyed playing with the toy train and learned to adequately master its controls. After Christmas, the train was put 
away. Afterwards, Thomas often asked to play with the train. But since it took so much effort to set up coupled with 
it needing an excessive amount of floor space in our tiny house, there was little incentive to bring the train out. I 
decided that the train might be a good fantasy context for Thomas based on his interest. My goals for this activity 
were very fuzzy, though two goals dominated my thinking early on. The first was that I wanted an activity in which 
Thomas would have to think several steps ahead to accomplish a task. Such ability to plan for the future is critical 
in problem-solving. Keeping one future task in mind while performing another task is trivial for most children of 
Thomas* age and adults but very difficult for Thomas. The second goal was lo continue to encourage Thomas to 
recognize meaning from abstract symbols. 

The activity begins by having a trolley-like train enter the track. The motion of the train can be controlled two 
ways: first, by changing the direction that the trolley will take when arriving at any of the several gates; second, the 
speed of the trolley can be controlled as well, though this function has never been recognized by Thomas. In Thomas* 
first experience with an earlier prototype, there was no clear goal to the activity other than watching the train go 
around the track. This was of little interest to Thomas. I quickly developed a second prototype, as shown in Figure 
4, in which I added "tunnels." We had played with makeshift tunnels with the real toy train at Christmas, so Thomas 
was familiar with the concept. In addition, he really enjoys going through tunnels when driving in a car. He first 
saw the second prototype by watching me first play the game. 1 immediately said that I wanted the train to go through 




Figure 4. Snapshot of the screen during Tom's Train. 
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the tunnel and he immediately found this goal interesting enough to want to pursue. 

I was amazed how difficult Thomas found the task of relating the abstract symbol of an arrow head indicating 
"future direction*' in this game. Fortunately, he persevered and continued to experiment, haphazardly at first. Of 
course, if one studies Figure 4 for a moment, it is clear that there are two gates that must be changed to get the trolley 
to go through the tunnel. Thomas finally succeeded in this fairly complex task. Interestingly, he will now choose 
this program, quickly make the trolley go through the tunnel, and then quit the program. This activity is ripe for 
adding other goals and tasks, such as picking up and dropping off passengers. 

Ghostbusters. This is the most recent activity I have developed for Thomas. Its goal is to find as highly a 
motivating context for Thomas to explore written language as possible. As per the activity's title, its theme is the 
movie of the same name and it was chosen simply because it is one of Thomas' most favorite movies. Using Apple's 
QuickTime video protocol, key scenes were digitized. The scenes I chose were based on my interaction with Thomas 
as he watched the movie — these were some of the scenes he found most memorable. All of the scenes but one arc 
highly verbal with dialog that Thomas "absorbed" from watching the movie, such as Bill Murray's "He slimed me!" 

When any of the four scenes are chosen, the entire scene is played, both video and audio. This is followed by 
the lines of dialog appearing to the side of the video window, which, when clicked, cause the video and audio 
corresponding to that line to be played. The other option is to interact with the scene through the slide bar at the 
bottom of the video window. The goal here was to provide a highly motivating reason to interact with the text lines 
again with the hope to promote the concept that print carries meaning. 

Interestingly, at the start Thomas seemed to avoid all of the printed words. In fact, though Thomas has played 
Ghostbusters for about three months he is just beginning to interact more frequently with the text. In contrast, he 
quickly learned (by the end of the first session) how to manipulate the QuickTime controls to play the scene over 
and over, frame by frame, and backwards. Again, he does not interact with this activity for very long, preferring 
instead to sample it for brief periods. So far, he is focusing on key phrases and his most favorite to date is Dan 
Ackroyd's "Actual physical contact!" 

There are still a few problems with the design of the interface. For example, Thomas gets confused about the 
need to press a "continue" button after viewing one scene to go back to the option to view any of the four scenes. 
The problem is with the software, not Thomas, and this needs to be changed. 

Adventures in The Playroom 

One of Thomas' recent birthday presents was another commercial product called The Playroom. The package 
has a series of activities which one chooses from the organizing graphic suggested in its title, a child's play room. 
The activities include simple math games, telling time, developing a "survival" sight word vocabulary, beginning 
letter sounds, a letter and word recognition activity resembling word processing, plus an assortment of other amusing 
activities. A particularly good feature of The Playroom is that it lends itsel f to creative exploration. One is rewarded 
by trying new things, though it is always easy to return to the "playroom" from an activity by clicking on the graphic 
of a "connecting door." Thomas' interest in The Playroom rivals that o\'KidPix\ he spends about 80% of his computer 
time between these two packages with the remainder spent on the software I custom designed for him. A special 
cartoon character called "Pepper Mouse" is a central figure in most of the Playroom's activities (a small stuffed 
version of Pepper Mouse came free with the software). Thomas refers to The Playroom as "peppa mus." 

There are too many activities in The Playroom to describe here, so only his favorite activity, "The ABC Book," 
will be briefly discussed. This activity uses beginning letter words (such as "U" for "unicorn") in a variety of fantasy 
contexts (such as a fairy tale castle, a barnyard, and a city street). When the user chooses any letter of the alphabet, 
the computer pronounces the letter and corresponding word and then "attaches" a picture of the word to the mouse. 
The user can move the picture around the screen and "deposit" it anywhere by clicking. Thomas spends extended 
periods of time interacting with this one activity alone. It has been among the most successful activities yet for 
Thomas in which language has been embedded in creative fantasy play. 

Thomas as Navigator 

One of the most interesting skills that Thomas has acquired is navigating effortlessly between software 
applications on the Macintosh desktop. It is hard to describe how this occurred and there is no magic "instructional 
plan" to share to make it happen with other children like Thomas. Over the course of the last two years, Thomas 
has just slowly assimilated simple strategies to achieve his "grazing" between all of ihc software at his disposal. 
Early after his cognitive breakthrough in understanding how graphical symbols could change the function of the 
mouse, he l>egan to pay closer attention to how I would intervene to solve unexpected problems. For example, when 
working with MacPaint, he would sometimes accidentally cause a dialog box to appear. The user is expected to make 
choices or changes and then click on an "OK" button, at which time the user is returned to the application. Thomas 
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apparently learned incidentally a rule like "When in doubt, click OK," by watching how I intervened to solve the 
problem. 

This example and others were probably due to my care in always verbalizing my actions even though I never 
forced Thomas to pay attention. I would talk through my "double -clicking," "Quits" and "Shut downs." Thomas 
eventually picked these up not because he was taught these techniques, but because he was part of the computer 
culture and eventually these became relevant and useful to him. These techniques became authentic tasks for 
Thomas because they were the "secrets" to controlling the computer for his own use. I am sure that Thomas' 
understanding of these techniques remains vague, but that is really beside the point. His understanding of "double- 
click to open" for example is closer to "click repeatedly until something happens." Fortunately, all Macintosh 
software reacts consistently to these techniques and rarely "punishes" users who do not have precise understanding. 
Being vaguely right is good enough. 

A Matter of Control 

It is difficult to distill features that comprise Thomas' most successful everyday experiences, though his time 
with the computer is the most prominent example. In fact, there are very few other experiences in which Thomas 
seems to exhibit the same level of satisfaction. Other examples include pouring himself a glass of soda, fixing a 
peanut butter sandwich, riding his bike at the park, drawin g with pens and markers, and imagi nati ve play with certai n 
toys. These are not general examples, but the "long" list of specific activities in which Thomas excels. The closest 
common denominator to these activities, I believe, seems to be that Thomas feels totally in control. He is expected 
and able to exercise complete freedom in how he chooses to explore these environments. 

When Thomas is working on the computer, he is in charge. His success in navigating around the Macintosh 
desktop to begin and quit activities has fueled the tendency to take other chances to learn new commands and 
functions. He has also been rewarded for his exploration. When working with the computer, I believe that Thomas 
considers himself to be combination of tinkerer, inventor, explorer, and trailblazer. This is his medium not only 
for expression, but also for risk taking. Another software package that Thomas has greatly enjoyed that has likewise 
encouraged a playful, exploratory attitude is the "living book" called Just Grandma andMe % in which the "Little 
Critter" book by Mercer Mayer comes alive through animation and sound on the computer. There is wonderful irony 
in knowing that Thomas finds comfort and confidence in a machine that often threatens and intimidates other 
normally functioning adults. 

Conclusions 

It is unclear whether any of Thomas' learning with the computer will transfer to other situations. For the 
moment, this does not seem very important. What is important is that he is learning and finding the computer to 
be an engaging medium. It is also important that all of Thomas' learning is necessarily incidental, though admittedly 
there is external intervention on my part to provide him with custom designed microworlds that will give him special 
opportunities to explore certain domains. Despite this external guidance, his learning remains incidental because 
he controls what he explores and what the goal of the interaction will be. He docs not play "Sour Apple," for 
example, to learn about simple counting and number theory, though why he does choose to play it is unclear. He 
may just enjoy watching apples disappear or find satisfaction in completing a task. 1 wish I had more insight to his 
behavior, but I am satisfied by his satisfaction. 

Alsounclearis where Thomas' abilities end and the computer's abilities begin as they interact with one another. 
I feel that their cognitive partnership is a concept worth reflecting upon. Thomas is engaged in what Salomon & 
Globerson (1987) have called "mindfulness" — deliberate, thoughtful, and challenging activity which requires 
much effort. Some might call this "work." Though the term has many negative connotations, those adults who love 
their work understand that there is little that can give them more satisfaction, despite how difficult it may be, no 
matter what occupation or profession it may involve. This is the kind of work that Thomas is engaged in. Whether 
or not Thomas will find a way to channel his mindful work to become an independently functioning adult in society 
remains to be seen. 

Ironically, Thomas's difficulties in meeting the expectations of society (including the educational system) is 
both his greatest liability and his greatest asset. On one hand, the people who matter most in Thomas's life are 
concerned about his future. As Thomas' parents, my wife and I worry about the possibility of his lifelong 
dependence on us. We also worry that we arc not giving enough attention to our daughter. Now that Thomas is 
i I years old, we realize it is a distinct possibility that he will never be able to function apart from us or professional 
caretakers.. Thomas' many classroom teachers over the years had have a wide range of concerns as well. Many, 
if not most, have worried about his lack of academic progress and socialization skills. The most well intentioned 
have focused on "survival" skills, such as crossing streets and grocery shopping, to increase Thomas's ability to 
function independently in society. The least well-intentioned have tolerated Thomas at best and belittled him at 
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worst, unashamedly bringing to the surface the extent of his differences. It is tempting to say that Thomas does not 
know how as yet to u fit in" to society and the classroom. However, it may be more accurate to say that society and 
the classroom do not "fit in" with Thomas. So, on the other hand, Thomas may not feel as shackled by the demands 
of society either because he is not aware of it, does not recognize it, does not comprehend it, or perhaps, does not 
accept it. Thomas's intellectual and social functioning still remains virtually ego centered and it is unclear whether 
he will ever be able to empathize with others. In some respects, this is a very enviable position. Society's demands 
have not reached Thomas as yet, and perhaps they shouldn't. 

The most recent progress Thomas has made has been social in nature. Slowly over the past year, he has showed 
clear signs of seeking other children his own age with which to interact. He is attending public school again and, 
so far, all is going very well. The reason for this is complex and multifaceted. Fortunately, several important 
attributes are working in Thomas' favor. First, I believe that he is seeking out interaction with other children. 
However, the nature of the interaction is quite fragile. Fortunately, his teacher's personality seems a perfect match 
for Thomas' plus she has clearly taken much time and effort to try to understand him. She deserves much credit for 
promoting aclassroom where mutual respect is the rule and for providi ng social and intellectual experiences derived 
from the children themselves. The cognitive "awakening" that Thomas is experiencing through his partnership with 
the computer is being matched by a similar social "awakening." He seems to be in the process of exploring his 
relationship with society. What seems to be most crucial is that he has been given a reasonable level of control over 
that exploration while being rewarded for taking initiative. Whether this turns out to be a trend or a temporary 
aberration remains to be seen. 

Finally, I have not stressed enough in this paper how much Thomas has taught me about design as I think this 
is a topic for discussion at a later time. I often remark in my design classes that working with children, especially 
young children, is among the best ways to learn about learner-centered design, even if one' s interest is primarily with 
adult populations. There are at least two reasons for this. First, young children are generally not able to bring to 
the learning environment as many strategies lo make up for poor design. Normally functioning adult users, if 
motivated enough, can bring al 1 types of learned cognitive strategies and prior knowledge to the table. Second, young 
children will not tolerate designs that arc poor, irrelevant, or meaningless, thus making for very quick feedback to 
the designer. Thomas has been aco-designer in all of the computer activities I have developed for him, even though 
he does not know it. The starting point for these custom built activities have been Thomas' interests and abilities 
combined with much speculation on my part about where he might be headed next. By involving Thomas in the 
design process at the beginning, he has largely determined the "instructional objectives." 

I end this paper with a simple story of success. Any small success is a great achievement with Thomas. Just 
recently, while eating out at a restaurant, Thomas read the word "exit," his first sight word besides his name. He 
immediately began talking about Pepper Mouse. 
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Footnote 

'Thomas had been having many unsuccessful experiences in the public schools. My wife and 1 came to the 
conclusion that the schools were failing to meet Thomas' needs and rather than try to "fight the system," we decided 
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it was in Thomas' best interest to home school him for a period of time. My wife assumed most of the home school 
responsibilities. We also enrolled Thomas in a Montessori school for a short time. This school seemed particularly 
well suited to Thomas given the nature of the Montessori method where students have much freedom to choose 
activities that interest them. In addition, Maria Montessori's methods were historically founded on her work with 
the mentally retarded. For two months* Thomas seemed to be doing well. Then, on the morning of the first day of 
the month, while Thomas was getting ready for school, the school's owner called to inform us thai, unfortunately, 
Thomas had been "dismissed" from the school and that he could no longer attend effective immediately. The owner 
worried about what effect Thomas' presence was having on the other children. Consequently, Thomas was home 
schooled for ihe remainder of that year as well. This example is not meant as an indictment of Montesorri schools, 
but it well represents the general attitude we have found among professional educators. There is much talk about 
inclusive enviionments, but little motivation and understanding to promote situations in which children with 
exceptionalities areconsidcredywrfoftheschool culture. This is unfortunate because normally functioningchildrcn 
have as much to gain as well. 
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Abstract: Level III interactive video is a flexible, hi-tech learning modality that has been 
utilized with both undergraduate generic and returning registered nurse students. Past 
studies have shown that level III interactive video can enhance education in a variety of 
settings with a variety of learners. Ongoing research into the effect of level III interactive 
video on nursing education is imperative. After a review of the literature, it was evident that 
there was a need to assess level III interactive video from the student's perspective. 
Therefore, a qualitative approach was indicated to determine the students' perceptions with 
regard to their learning needs, personal needs, and ability to cany out the nursing process. 
A pilot study was conducted and the interview schedule was revised. Each subject will 
complete the same level III interactive video module and interview schedule. This funded 
research is in process. 



Introduction 

Advances in instructional technology have made it possible to utilize interactive video in the nursing 
education arena to enhance and augment clinical nursing experience. Given the current state of affairs in 
nursing education, limited clinical experience with decreased access to diverse clients, interactive video 
appears to be the answer for the future. Interactive video combines the power of the computer with the 
visual and/or auditory capabilities of the videodisc. The literature has portrayed interactive video in a 
favoraUc light from an educator-based perspective (Lanier, 1986; Kozma, 199C; McNeil & Nelson, 1990), 
However, before we begin to integrate interactive video throughout nursing curriculums, it is imperative to 
determine its effectiveness from the learner's perspective. 

Andragogical, or adult learning, principles apply to nursing students; as adults, they build on and 
relate to past experiences they have encountered. To assess the potential of interactive video with the 
adult learner, one must recognize that not all adults are the same. The adult education literature supports 
the view that there are levels of difference within age groups of adults (Brooklield, 1990; Darkemvald & 
Merriam, 1982). Whereas andragogy provides a conceptual framework stating that all adults require high 
levels of input into content that is relative to their needs, course designs that are conducive to learning, 
and evaluation of their learning needs and the activities associated with learning. It is still a question as 
to how early in the learning process various adult learning groups should be involved with extensive 
input. The generic nurses are typically a younger population, often having familiarity with computers and 
communication teclinology. The returning registered nurse students arc a comparatively older group, 
typically with high motivation and a rich "reservoir of experiences'* (Knowles, 1980, p. 45). In the future, 
for instance, can we anticipate that the high comfort level of generic students with technology will make 
them ready candidates for interactive video? Will it be the case that they will need more repetition and 
structure through the interactive video due to their more limited experiential base? On the other hand, 
will returning registered nurse students respond more enthusiastically to interactive video due to their 
high level of motivation and wealth of experience or will they first need to receive hands-on computer 
training? Which group will respond better early, which group will respond better later, and what 
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e " strategies will the field need? From an andragogical stand point, interactive video is a learner-driven 
educational modality; the learners access and view learning segments based on their learning needs. The 
issue to be investigated is how well the two groups will respond to the andragogical capacity of interactive 
video and the extent to which the field needs to be sensitive to the addition of this technology with these 
two groups of students. 



The purpose of this study is to determine the extent of andragogical effectiveness of interactive video 
from the learner's perspective. The learners will be interviewed according to andragogical criteria to 
determine if their learning needs have been met. These criteria are learner-driven and include whether 
the learning environment, activities of learning, and participative learning, through learner control of the 
interactive video presentation, met their needs. This proposal suggests a qualitative descriptive study is 
needed since it is important to determine the learners 1 perspectives regarding this innovative instructional 
technique. This study will provide learner-based information that nursing educators, who deal with adult 
learners, need in order to assess the integration of interactive video into entry level nursing education 
and/or professional continuing education for nurses. 

The specific aims of this study are: 

1) To delineate the undergraduate nursing students 1 perceptions of instruction via interactive 
video versus traditional classroom lecture format. 

2) To identify whether the visual presentation contained within the Interactive video enhanced the 
textual format for purposes of learning on the part of the student. 

3) To identify students 1 needs (learning, personal) that are met with the use of the interactive video . 
instruction and may not be met by traditional classroom lecture format, based on the learners' 
perceptions. 

4) To determine whether the Interactive video learning modality enhanced students' sense of 
competence with respect to canying out the nursing process, based on the learners' perceptions. 

5) To contrast generic nursing students 1 and registered nurse students 1 perceptions of the effectiveness 
of interactive video from the learners' perspective with respect to andragogical criteria. 

6) To make specific recommendations to the field on the effectiveness, desirability, and use of 
interactive video with generic and registered nurse students. 

The long term goal of this qualitative descriptive study is to provide a basis for designing a model for 
integrating clinical components in nursing education via interactive video. This model will be utilized to 
develop more comprehensive studies focused toward designing and testing an intervention that nursing 
educators can implement to facilitate clinical nursing education. 



Nursing education is changing. It is a challenge to provide students with the clinical experience 
necessary to meet their learning needs. As acute care settings become more specific, the students' 
exposure to diverse clients during their clinical rotation is decreasing. This limited experience necessitates 
the use of innovative teaching techniques such as interactive video. Interactive video combines the power 
of the computer with the visual and/or auditory capabilities of the videodisc. According to Sweeney and 
Gulino (1988), "the videodisc brings to the screen clear views of the patient, the equipment, or the setting, 
rather than relying on the printed word or computer generated graphics to get the message across" (p. 
158). Interactive video is self-paced and learner controlled. Based on their learning needs, the learners 
control their own interactive video experience by utilizing the computer. Built in navigational controls 
facilitate the students' individualized movement throughout the learning module. According to Poguc 
(1988), "learning theory supports the premise that the major advantages of the computer as a tool of 
instruction arc its ability to allow us to provide interaction and individualization as an integral part of the 
instructed process" (p. 95). Roberts (1981) contends that most adults do best in self-pacca situations given 
the internal nature of learning. Gaycski and Williams (1985) have identified four levels of interactive 
video; for the purposes of this study, interactive video will refer to Level III interactive video. Level III 
interactive video involves constant angular velocity (CAV) format videodiscs controlled by an external 
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Press computer connected to a commercial grade laserdisc player (Gayeski & Williams, 1985). The computer 
portion of the program can readily be rewritten, or repurposed, for a specific instructional use. Most 
interactive videodiscs on the market today are Level III applications. 

Interactive video presentations have been shown to be well received by people with no familiarity with 
computers and in various professions and occupations: dairy fanners (Debloois, 1981); military personnel 
(DeBloois & Woolley, 1981); white collar workers (Main, 1984). The literature remains mainly educator- 
driven (Kozma, 1990; McNeil & Nelson, 1990; Lanier, 1986), Kozma (1990) described interactive video 
as cost effective in materials and personnel, particularly when safety is a factor in learning content of 
disciplines like chemistry or health. According to the work of Lanier (1986), "interactive video is time 
efficient; the user can go through the program at his or her own pace...Learning specific tasks takes an 
average of iO-35% less time with the disc than with older methods. Retention of material is better" (p. 
37). In a meta-analysis of interactive video instruction research of the past 10 years, McNeil and Nelson 
(1990) noted an overall mean achievement effect of 0.530, indicating that interactive video is an effective 
form of instruction. The literature also supports the belief that interactive video aids in clinical decision 
making (Clark, 1984; Sweeney & Gulino, 1988; Weiner, 1988; Hodgin, 1986). According to Brown and 
Minton (1993), Classroom 2000 is a state-of-the-art technology project to enhance employability skills. 
They contend that, "by repeated consideration of the behaviors and attitudes appropriate in the workplace, 
students learn to understand how employers think and how, as workers, they must behave on the job in 
order to retain employment" (p. 67). Clark (1984) reported that interactive video instruction, utilized by 
the American Heart Association, trained students in CPR in one-third of the time needed with the 
conventional method and the retention rate doubled. These projects (Clark, 1984; Brown & Minton, 
1993) demonstrate the impact technology can have on the cognitive, affective, and psychomotor learning 
domains. It is easy to extrapolate the use of technology, such as interactive video, into nursing education 
since nursing practice encompasses all three domains: cognitive, affective, and psychomotor. Nursing 
education is an area in which interactive video could be utilized to enhance, augment, and/or replace 
traditional forms of instruction (Kozma, 1990). Sweeney and Gulino (1988) state that, "microcomputers 
offer tremendous potential to the clinical educator. They are especially suited for teaching applied skills 
when coupled with a videodisc player" (p. 158). Interactive video had been described as most effective in 
providing challenging simulations that aid in developing problem solving abilities and decision making 
skills (Hodgin, 1986; Weiner, 1988). As clinical objectives become more difficult to meet for every 
student, the high-tech, flexible, learning modality of interactive video could be utilized to provide all 
students with the same needed information and experiences. The interactive video learning modality 
facilitates the learners 1 grasp of the material while increasing their level of retention (Clark, 1984). 

According to Dover and Boblin (1991), it is necessary to prepare nursing students for the ever 
changing health care environment. Maternal/child health nursing is a challenging area for nursing 
educators; the time spent in labor and delivery is limited and the variety of births observed is not 
predictable. This project would utilize interactive video to review and expand the students' encounters 
with birth experiences. This learning modality is applicable to both the registered nurse and the generic 
undergraduate nursing student (Sweeney & Gulino, 1988). Nursing is a profession based on practice 
encompassing cognitive, affective, and psychomotor skills. The nursing students must be able to assess 
and cognitivcly and affectively create a holistic composite of their clients from which to draw conclusions 
or formulate nursing diagnoses, goals and objectives. They must then choose and, using their 
psychomotor skills, carry out appropriate nursing interventions to facilitate the client meeting the 
established objectives. They then evaluate both the effectiveness of the nursing care and the client's 
progress toward the designated goals and objectives. Interactive video would address the learners' needs 
for application driven content as well as the andragogical criteria which are believed to facilitate and 
enhance adult learning. According to Knowles (1980), "the way to produce competent people is to have 
them acquire their knowledge (and skills, understandings, attitudes, values, and interests) in the context of 
its application" (p. 19). This instructional technology enables the students to work through actual client 
situation simulations via videodisc and permits 'safe' problem solving outside of the clinical setting. This 
promotes problem-solving abilities and facilitates clinical competence. Bandura and Cervone (1986) feel 
that as students extend their powers of critical thought there is an increased sense of self-efficacy and self- 
worth. They state that these feelings facilitate learning and the ability to apply it. 
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The conceptual framework for this research project arises from Knowles (1980; 1984) theory of 
andragogy that both empowers the learners 1 selection of relevant content and enhances self-directedness. 
Knowles defines andragogy as M the art and science of helping adults learn M (1980, p. 43). According to 
Knowles, it is the foundation for developing the competence to engage in lifelong, self-directed learning. 
He describes seven phases in the andragogical process: M l) establishment of a climate conducive to adult 
learning; 2) creation of an organizational structure for participative learning; 3) diagnosis of needs for 
learning; 4) formulation of directions of learning (objectives); 5) development of a design of activities; 6) 
operation of the activities; 7) rediagnosis of needs for learning (evaluation)" (p. 59). Knowles contends 
that learning is an internal process involving the total person. He describes the learning-teaching 
interactions as the recognition of adult students 1 needs and the development and use of superior principles 
of teaching. Learners accumulate an increasing supply of experience that serves as resources for their 
own learning. The learning process is related to and makes use of these experiences with the learner 
actively participating. The critical function of the teacher is to create a rich environment in which 
students' motivation to learn is enhanced since the critical part of learning is the interaction between the 
learner and his or her environment. According to Williams and Benedict (1990), nursing faculty 
members must "recognize their students 1 rich and diverse knowledge and experience in designing learning 
experiences which are consistent with adult learning theory" (p. 201). The principles of andragogy must 
be integrated into nursing education in order to enhance nursing students' learning experiences. Using 
interactive video enables the instructor to create and/or choose multimedia learning environments in 
which the students' can interact. The student interaction is based on the students' needs for information 
and can be individualized by the choices they make. 

Our study is designed to determine the andragogical learning conditions of interactive video from the 
students' perspective. The assumptions are that the learners' learning needs as well as their personal 
needs will be met. The learning needs of the student are addressed in this modality since the navigational 
tools provided permit them to view any portions of the learning module they choose. This facilitates 
entry-level adults or generic nursing students as well as returning registered nurses continuing their 
education. The generic nursing students' learning is augmented by the use of interactive video. This 
modality could maximize their educational experience by facilitating the integration of theory and clinical 
practice skills and greatly enhance their educational experience (Sweeney & Gulino, 1988). Using 
interactive video would provide the returning registered nurse students with the opportunity to review 
clinical components previously experienced in their basic nursing programs while synthesizing the 
theoretical concepts of their baccalaureate education. Adult students' personal needs for self-paced, 
flexible, instructional modules should be easily addressed with the flexible, hi-tech modality of interactive 
video. 



Research Design 

This qualitative descriptive study will be carried out by means of one semi-structured interview 
consisting of open-ended questions to explore the perceptions of nursing students using interactive video. 
Rizzolo (1990) conducted a Delphi study exploring the factors influencing the use of interactive video. 
Rizzolo concludes, "the place of [interactive video] in all educational settings has not yet been 
determined" (p. 157). Kozma (1990) belicxcs that interactive video is particularly beneficial in the 
discipline of health education where safety is a factor. There have not been great strides made since 
Haanafin (1985) recommended that learner control and the positive attitude toward interactive video be 
explored. This descriptive approach facilitates the data collection with a minimum of investigator bias 
while permitting the nursing students to express their feelings toward interactive video (Berg, 1989). 

This study is designed to answer five research questions: 

1) What are the perceptions of undergraduate nursing students with respect to the interactive video 
learning modality versus traditional classroom lecture format? 

2) Are the visual presentations contained within the Interactive video learning modality enhancing to 
the textual format as well as to student learning? 

3) What needs (learning, personal) of the students arc met with the use of the interactive video 
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learning modality? 

4) What is the effect of the interactive video learning modality on the students 1 sense of competence 
with respect to carrying out the nursing process? 

5) Are there qualitative differences between the generic nursing students' and the registered nurse 
students 1 perceptions on the effectiveness of interactive video? 

Sample 

This opportunistic and purposeful sample will consist of six generic undergraduate nursing students 
from the University of Pittsburgh and six registered nurse undergraduate students returning for their 
baccalaureate nursing degrees at Penn State University. In addition, one generic nursing student and one 
registered nurse student will participate in the pilot phase of this study. The descriptive nature of this 
study and its method of inquiry support the adequacy of a sample size of twelve. Qualitative inquiries, 
according to Lincoln and Guba (1985), need only be large enough "to detail the many specifics that give 
the context its unique flavor" and "to generate the information upon which the emergent design can be 
based" (p. 201). 

Measu res/Instruments 

One semi-structured interview consisting of open-ended questions will be employed to facilitate 
discussion regarding interactive video and traditional classroom lecture format from the perspective of the 
nursing students. The interview schedule will be loosely derived from the literature and will be evaluated 
for face and content validity by experts in the areas of qualitative research and interactive video. In 
addition, the interview schedule will be piloted on two undergraduate nursing students, one generic 
nursing student and one registered nurse student. 

Procedure 

Each participant will complete the interactive video program, Perinatal Family Care: A Learning 
Module on Intrapartal Nursing Care. The semi-structured interview will be scheduled within two weeks 
of the participant's completion of the interactive video program. The interactive video and interview 
schedule will be tested on two undergraduate nursing students, one generic and one registered nurse 
student. After revising the interview schedule, appointments will be made to meet with each volunteer to 
obtain written consent for his/her participation in the study, answer any and all questions he/she may 
have, confirm the audio taping of the interview, review confidentiality measures, and schedule their 
interactive video session. Once the interactive video session is arranged, the interview will be scheduled. 

Plan for Data Analysis 

Reliance on the transcribed semi-structured interviews, rather than fixed-choice instruments, permits 
openness to features of andragogical processes that may not be anticipated in advance. Consequently, the 
data will initially be somewhat unstructured. The necessary structure for data analysis will, however, be 
provided during analytical procedures which will include organizing, coding, categorizing, and 
generating themes and patterns, The transcribed interviews will be analyzed by the research team 
employing content analysis techniques prescribed by the literature (Polit & Hungler, 1983; Fox, 1982). 
The inter-coder reliability formula specified by Miles and Huberman (1984) will be used to compare the 
data analyses of the research team members. Reliability of data analysis will also be enhanced and made 
less cumbersome with the use of Ethnograph (Seidel, Kjolscth, & Seymour, 1988), a computerized system 
for coding, filing, retrieval, and revicv\ Four of the study participants, two generic nursing students and 
two registered nurse students, will be cortacted to read and validate the teanVs findings. An aggregate, 
narrative description of the interactive video experience, derived from the content analysis process, will be 
given to the students for verification. 
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Abstract: Universities across the nation are stressing the 
importance of integrating multicultural experiences into 
the classroom. Faculty at Florida State University and 
Colorado State University have developed three creative 
ideas incorporating the use of computer-aided design and 
multicultural experiences into their computer aided design 
(CAD) courses. As a result of these projects, student 
appreciation of diverse cultural influences increased and 
design solutions were improved. 

Introduction 

Universities across the nation are stressing the importance of integrating multicultural experiences 
into the classroom. Within interior design and housing programs, many innovative ideas are developing as a 
response to this demand. At Florida State University and Colorado State University, design faculty have 
developed three creative ideas incorporating the use of computer-aided design (CAD) and multicultural 
experiences into the CAD lab. Although these projects could be incorporated into any design course, CAD 
systems allow students to develop a motif/image, store it as a symbol, and insert or manipulate it as desired. 
This paper will discuss the multicultural projects, the benefit to the students and tta enhanced results of the 
experience. 

Background 

International concerns along with multicultural sensitivity are of great concern to business and 
education today. Guerin (1991), in an article addressing the future of design education, lists international 
concerns as one of the top issues facing design education in the 21st century. She states, "contextual design 
with cultural sensitivity is an important component of interior design education." Henry Ferguson (1987), 
explains the importance of learning about ones own culture and how that relates to acceptance of other 
cultures. He states, "by knowing about his culture, one becomes intellectually and spiritually free." Naisbitt 
and Aburdene (1990), point out the although we may be evolving into a world economy, there are 
unmistakable signs of a powerful countertrend: " a backlash against uniformity, a desire to assert the 
uniqueness of one's own culture and language". Professions that interact internationally, or have 
international clients, such as interior design, must emphasize intcrcultural sensitivity, awareness, and 
knowledge about foreign establishments (Guerin, 1991). 

Preparing professionals who can address the needs of a culturally diverse population is a significant 
national challenge for all educators. While it is still on the periphery rather than integrated to die curriculum 
in most schools and colleges, multicultural education has made significant inroads in curricula within the last 
two decades (Banks, 1993). Some of the nations leading colleges and universities have either revised their 
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general core curriculum to include ethnic content or have established an ethnic studies course requirement. 
For example, Florida State University requires all freshman to take six semester hours of multicultural 
courses (FSU Bulletin, 1991-92). However, Fferguson (1987), argues that when multicultural studies are 
conveniently relegated to "World Culture" classes, students view them as "beside the point". At a workshop 
at the University of Florida, Rentier (1989) mentioned the difficulty in persuading faculty members to 
internationalize their existing courses and feels the international dimension should be a natural part of any 
well-conceived major. Commitment to improving diversity at Colorado State University has high priority 
and is evident in the multitude to activities conducted to improve diversity (CSU Progress Report, 1991). In 
essence, multicultural education is a movement designed to restructure educational institutions so that all 
students, including middle-class whites will acquire the knowledge, skills, and attitudes needed to function 
effectively in a culturally and ethnically diverse nation and world (Sleeter & and Grant, 1988). 

Multicultural education is not an ethnic or gender specific movement. It is a movement designed to 
empower all students to become knowledgeable, caring, and active citizens in a deeply troubled and 
ethnically polarized nation and world (Banks, 1993a). Five dimensions to multicultural education have been 
defined as l)content integration, 2)knowledge construction process, 3)prejudice reduction, 4)equity pedagogy, 
and 5)and empowering school culture and social structure (Banks, 1993b). 

The typical United States citizen has become difficult to describe. By the end of the 20th century, 
the Hispanic population will have increased by an estimated 21%, the Asian by about 22%, the African- 
American by almost 12%, and the white by a little more than 2%. Census statistics document a trend in 
population growth in which by 2020, Hispanic or non-white U.S. residents will have doubled and the white 
non-Hispanic population will not have increased at all (Henry, 1990). Immigration to the U.S. has steadily 
increased since 1960 and now exceeds one million per year (Statistical Abstract of the United States, 1992). 
In contrast to these major demographic changes, students attracted to the interior design professions continue 
to be predominately white female (ASID, 1992). 



Recognizing the importance of having diversity incorporated into existing classes and academic 
majors, three design projects were developed for use in two FIDER accredited interior design programs at 
Colorado State University and Florida State University. The goals of the projects were to expand the 
student's use of the computer, to study various cultures of the world and to heighten the student's awareness 
of how diverse their personal ancestral cultures are and how they contribute to the diversity of their 
university environment. This was to be accomplished without sacrificing the quality or altering the goals of 
the existing course. The details of projects are discusses below. 

As a beginning CAD assignment, students were given the choice of creating a custom design for a 
door or entryway detail, a fabric, or an area rug. Three different experiences were developed with the 
emphasis on studying history and culture: a visit to the Aztec Exhibit in Denver, Colorado; researching 
Seminole Indians in Florida, and student research into their personal ancestry. 

Colorado State University students were offered the opportunity to view the Aztec exhibit in 
Denver. Through the generosity of the Mexican people, nearly 300 artifacts which dramatically illustrated 
the rich cultural legacy of the Aztecs was shown in Denver, Colorado. At that time, with the North 
American Free trade agreement close to passing, it seemed appropriate for design student to study and 
appreciate the heritage of their neighboring country. 

Florida, State University students had the opportunity and the responsibility to learn about Seminole 
Indians. FSU uses the name Seminole and the school "mascot''. The instructor felt strongly that students 
should not graduate without understanding who the "real" Seminole Indians where. In addition, the 
Seminoles have beautiful patchwork patterns that are used when they create clothing. These patterns are 
varied and made an excellent source of inspiration for students. 

Both Florida State and Colorado State students were also given the opportunity to research their 
personal ancestry. Many students were well-informed regarding their own history, but others had a lot to 
learn. Students were encouraged to research back as far a four generations if possible. In all cases, cultural 
artifacts were gathered and incorporated into their custom designs which were developed on the computer 
using AutoCAD software. Students located common symbols and images from these cultures such as art, 
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architecture, costume, textiles design, or other visual artifacts. 

Students then used the computer to create, manipulate, and print their designs. 
AutoCAD drafting and design software contains a number of useful command that allow students to explore 
many design possibilities. The editing commands are especially useful for creative problem solving. The 
move, copy, array, mirror, scale, trim, and extend commands are just a few of the many editing options that 
allowed students to explore design alternatives. The design process is interesting to observe when students 
are creating using the computer. Students were encouraged to sketch while visiting the museum or library. 
From those sketches, students can begin to create and manipulate images on the screen. The create of 
blocks or symbols allows entities to be created and then inserted, rotated, and scaled. At Colorado State 
drawings are first created in AutoCAD, then transported to Animator Pro to add color, £ °-n to 35 mm slides 
as final output. At Florida State, students create two designs, the second being a variation of the first. This 
allows them to see the flexibility and adaptability of software. In addition, after researching their chosen 
topic, they wrote a 2-3 page summary detailing the source of their inspiration for the project The paper 
enhances the reliability of their sources of information and discourages stereotypical designs that may have 
occurred without thorough research. Designs were then presented to the class with class discussion 
encouraged. 

Summary 

Through these assignments and activities, students increased their knowledge of design while learning more 
about their world. Specific results from the CAD projects showed a greater variety of designs and a 
richness and depth to the designs. This depth and richness had not been seen in previous semesters when 
the cultural research was not required. In addition, the class discussions during presentation were much 
more detailed and research-based than those in previous semesters. Following the presentation, students 
were asked to react to the presentations and recall something they had learned during the presentation. All 
students reported they had learned something they did not know about another culture and a majority learned 
more about their own backgrounds. As a result of these projects, student appreciation of multicultural 
influences increased and design solutions seemed to be a better quality as a whole due to the required 
research. University goals were successfully achieved by looking to the past and developing an appreciation 
of the diverse future. 

In order to develop tools necessary to function productively in a pluralistic society, educators must 
move beyond awareness of multicultural differences arid technical information to application of knowledge 
with diverse groups (DeVaney & Hughey, 1992). Students awareness of their personal multicultural 
differences is the first step to effective multicultural education. 
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ABSTRACT Los Alamos National Laboratory (LANL) developed a model for 
school networking using Los Alamos Middle School as a testbed. The project was a 
collaborative effort beiween the school and the Laboratory. The school secured 
administrative funding for hardware and software; and LANL provided the network 
architecture, installation, consulting, anc training. The model is characterized by a 
computer classroom linked with two GatorBoxcs and a UNIX-based workstation 
server. Six additional computers have also been networked from a teacher learning 
center and the library. The model support infrastructure includes: local school system 
administrators/lead teachers* introductory and intermediate hands-on teacher training, 
teacher incentives foi involvement and use, opportunities for student training and use, 
and ongoing LANL consulting. Formative evaluation data reveals that students and 
teachers alike are finding the Internet to be a tool that crosses disciplines, allowing 
them to obtain more timely information and to communicate with others more 
effectively and efficiently. A lead teacher's enthusiastic comments indicate some of 
the value gained: "We have just scratched the surface. Each day someone seems to 
find something new and interesting on the Internet. The possibilities seem endless." 



h PROJECT BACKGROUND 

Motivation 

The world of computer networking is opening doors to vast and wonderful classroom resources. Networking 
technology has advanced rapidly, and networking in the schools has been and, it appears, will continue to be 
implemented at an increasing rate. Five years ago, little potential existed for using internetworking in schools. 
Now, in 1994, the advantages of using networking technology in education are readily apparent. Libraries, 
' irsities, national laboratories, corporations, companies, and classroom teachers are sharing or making 
info>\r<«uion and resources available through computer networks. The potential for our students is limited only 
by imagination. As Newman (1993) stated, "computer networks hold the key to the large-scale implemenation 
of school technology." 

A national goal is to have every public school connected to a national information highway by the year 2000. 
Is the nation prepared to network all public K-12 institutions in the United States within the next six years? To 
meet this national goal, we must know what is working and what is not working about hardware, networks, 
software, curriculum integration, structuring issues, maintenance, and support. Wc must document the lessons 
learned to avoid reinventing the wheel and to leverage federal and stale money effectively. 
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History 

In early 1993, Los Alamos Middle School (LAMS) used an existing Apple Macintosh-based computer lab with 
a local area network (LAN) to begin a new technological effort. Mr. Trottier, the LAMS principal, discussed 
with Los Alamos National Laboratory (LANL) the possibilities of designing a new network and obtaining help 
with connection to the Internet. 

In 1993, LANL staff assisted LAMS teachers in this effort. The Los Alamos School District additionally 
supported this project by providing the funding to purchase equipment and software. 

Many schools are becoming "connected;" however, being able to use that connectivity effectively is difficult for 
classroom teachers. Connie Stout of T-Net, a Texas networking effort, summed it up very well when she said: 
"Fear of the technology, inadequate funding and support, lack of adequate reasons to use the network, lack of 
access to telephone lines, and lack of time in the day are just a few of th* hurdles that must be overcome before 
these networks will be used by PreK-12 educators." (Apple Classroom of Tomorrow Project) 

There is a clear need not only to make networking technology and information available to the schools but also 
to provide training, software tools, and interfaces to make that technology and information readily and easily 
usable by our educators. LANL staff and LAMS teachers in this effort all agreed that the technology installation 
was just one part of the integrated educational technology picture. Instruction about available resources, how to 
access them, and how to use them easily and effectively is as important Backed by the principal and supported 
by the lead teachers, we decided that training sessions should be developed to provide instruction for all the 
teachers and staff at the middle school in the use of the new technologies and the Internet. LANL staff members 
designed two instructional sessions (delivered multiple times), provide consulting support, and arc planning 
additional training sessions. The LAMS lab directors/lead teachers are also designing and teaching additional 
sessions and providing lab time and assistance for the teachers. The results have been very rewarding, and 
feedback from the teachers has been very positive. 

We want to share what we have learned from the middle school project with other schools and people interested 
in school networking, connecting to the Internet, or using Internet connectivity to a better advantage. 



II. A MODEL FOR SCHOOL NETWORKING 



Goals 



A goal has been set by the federal government for all schools to be connected to a national network within the 
next six years. Yet, there are very few documented, feasible, and validated models of school networking for 
administrators and planners to follow. The goal of this project was to establish such a model, building upon 
both school resources and national resources in a partnership arrangement Primary goals were to 

• explore how we can connect the schools to the Internet in a scalable manner; 

• document the model for use by other schools; and 

• use the middle school as an educational networking testbed to determine 

- factors involved in connecting a school; 

- ways to help students, teachers and administrators use the resources effectively; 

- ways to train teachers to be system managers; and 

- steps for helping connected schools to become self-sufficient. 

Success Factors 

Major keys to the project's success were as follows: 

• Significant, dedicated, continuous administrative support (the principal) 

• Dedicated, qualified school computing staff that supported networking (lab teachers) 

• Adequate, state-of-the-art hardware and software provided by the school system 

• Facilities dedicated and supported for computer networking (existing lab) 

• Technical support for start-up, network design, and training (LANL staff) 
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Characteristics 

Figure 1 is a schematic of the LAMS model networking project. Distinguishing characteristics of this model 
include the following catergorized features. 

Technical Design: 

• Central computer lab, with additional satellite workstations throughout the school 

• Central, UNIX-based server accessed only by the school's system administrators 

• Liberal use of public domain software 

• Security plan 

Strong, Coordinated School Approach 

• Ongoing support by the school administration 

• Dedicated and qualified school staff supporting computing and networking 

• A financial plan and proposals for hardware and software acquisition 

• Collaboration with parents knowledgeable in the relevant technologies 

• Request for state-of-the-art hardware and network architecture 

• Incentives for teachers to learn about and use computer networking resources 

User Support 

• Training program for lead teachers 

• Training workshops for all school teaching and administrative stall 

• Computer lab scheduling to allow usage opportunities for students and teachers 

• Consulting support 

Monitoring and Evaluation 

• LANL monitored the effectiveness of the teacher training sessions and the school's overall use of the 
Internet The. evaluation took the form of direct observation, interviews with students and teachers, 
examination of curricular applications and output (projects that used Internet information), and 
questionnaires. 

Ill, DESIGN AND 1MPLMENTATION 

Equipment And Connectivity 

The LAMS computing environment is comprised primarily of Apple Macintosh (Mac) computers connected by 
an Apple LocalTalk network to several printers. Several models of the Macintosh exist within the LAMS 
environment, ranging from Classics to a Centris 610 with the most common model being the LC111. We found 
that the tool, InterPoll, from Apple worked well to survey machine types, system versions, and printer software 
versions. 

The main computer lab was divided into two networks because there were too many systems to be connected on 
a single LocalTalk network. Each network has a GatorBox that acts as a gateway for a PhoneNct link to the 
Ethernet All systems were upgraded with MacTCP (version 2.0.2) to insure that system software would work 
with MacTCP (Macintosh Transmission Control Protocol). 

Configuration of each Macintosh was tedious but straight forward. Each system needed a unique system name, a 
default uscmamc, and an IP (International Protocol) address. Specific parameters, such as the gateway and name 
server addresses for MacTCP, were set on a default version that was loaded on each system. Therefore, only 
those parameters unique to each system had to be entered manually. 

In this particular situation, the local high school was already connected to the Internet; so we needed only to 
provide a connection from the middle school to the high school. This connection, a basic rate Integrated Service 
Data Network (ISDN) line, was provided by USWEST, the local phone carrier. LANL staff specified the ISDN 
line, hooked up the line to both LANs at the two schools, and performed troubleshooting of the ISDN line and 
the connections. The middle school ordered the ISDN line from USWEST and specified where the line was to be 
installed. 
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Soitware 

An essential step was to determine the requisite software for the server and find software to support the goals of 
the project for the Macintosh clients, Most of the required software was freely available over the Internet, The 
project initially required software that would access Gopher servers, World Wide Web (WWW) servers, network 
news servers and also facilitate electronic mail (e-mail) connectivity. Some support software was also needed. 

The next step was to install the requisite software on the server. Software servers set up on the UNIX-based 
workstation included: POP (Post Office Protocol); DNS (Domain Name Service); Gopher, enhanced 
anonymous FTP (File Transfer Protocol) service; and WWW. 

Public domain and commercial software packages were selected, obtained, and installed. Public domain software 
obtained over the Internet to meet project goals included: TurboGopher, NCSA Mac Mosaic, News Watcher, and 
Eudora (with support utilities Stufflt Expander, JPEGview, Sparkle, Ulaw, UlawPlay, SimplcPlayer, Image, 
DownLinc, and NCSA/BYU Telnet). 

Commercial packages purchased by the school were also installed and included: Apple MacTCP, Apple 
Quicklime, and VersaTerm Network Utilities 

Other public domain software packages loaded for specific projects included: BinHex 4.0, Compact Pro, Digital 
Camera, InterNews, InterPoll, GradeBook, Kolor 2.0, MacWAIS, MoviePlayer, SndPlayer, Talk-2-me, Tidy it 
up, Ultra Recorder, and MacTCP Ping. 

Issues *nd Problems 

Equipment: The use of a UNIX-based server can be seen as both a strength and a weakness of this model. The 
strength is that it enables us to easily provide network services such as DNS, POP, and other servers using 
public domain software. The weakness is that UNIX and its administration are nontrivial to learn; therefore, 
nearby experts who am willing to assist on an ongoing basis at the beginning of the effort are essential. The 
robustness of the UNIX server and the proximity of experts at LANL made this a cost-effective testbed approach. 
As the technologies improve and the number of UNIX experts increases, this weakness will probably disappear. 

The Macintosh is designed to be a "personal" computer; it assumes that only one person will be using the 
system. Specifying a "usemamc" presented a problem because these systems were in a computer lab and would 
be used by many students and staff members. Therefore, we adopted the approach of specifying a generic user 
for each system; for example, a system named "Mac-12 M had a corresponding uscrname of "User-Mac-^." For 
most of the applications, having a generic username presented no problem. However, for e-mail, this created a 
major hurdle to overcome. Luckily the solution was simple. The e-mail software was not installed on any of 
the systems in the computer lab. Each user was given a diskette that contained Eudora, the e-mail utility, and a 
unique settings file that identified them as the user. With the diskette, users could retrieve and archive mail, and 
they could move freely between systems while preserving the privacy of their e-mail. 

Software: Particular attention was paid to software versions. This was especially true of system software such 
as MacTCP. We were careful to use the latest version of software and ensure that it was compatible with the 
operating system on each Macintosh. Most systems were running version 7.1 of the MacOS; however, a few 
older machines were running earlier versions. 

Furthermore, keeping the software base current with the latest release of each application is not trivial It is 
important that the systems users have a consistent software environment. New software must be introduced 
carefully to lessen the impact on these mostly novice users. We have yet to work out a standard procedure for 
the introduction of software upgrades. 

System Clocks: Once systems are connected to one another in a LAN, synchronizing their internal clocks is 
important. As you move software from system to system in the network, having an accurate time associated 
with files is important so that you know which file is the newer version. We used VersaTerm Time Client as a 
utility to deal with this problem; it updates the system clocks across the network from a network time server. 
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Communications Lines: We had major problems getting the ISDN line installed correctly. Most of the 
problems were delays; it took a long time to gel the line installed. Once the line was installed, many phone 
calls were necessary to troubleshoot the line and the local carrier's switching equipment. 

IV. TEACHER TRAINING AND INTERNET USE 
Lead Teacher Training and Support 

When it became apparent that the installation of the Internet connection and LAN was imminent, two lead 
teachers were trained as the system administrators. These lead teachers were to get a head start in learning how 
to access and effectively use Internet software and information. The lead teachers served as system 
administrators, learning to use UNIX on the server, and as Lab assistants during initial teacher training hands-on 
workshops. Now they arc Internet teachers for classes of students and also provide individualized instruction and 
consulting for teachers, students, and staff. 

Teaching and Administrative Staff Training Sessions 

Two teacher training sessions were designed and held for all LAMS teachers and administration staff. In the first 
session, teachers learned to access information through Mosaic and Gopher servers and get net news using 
NewsWatcher. In the second session, they learned about e-mail using Eudora. In these sessions, the staff worked 
in pairs for one-hour periods listening to descriptions of tools and capabilities and then using the tools hands-on 
in the lab. 

Evaluation results revealed that most of the teachers agreed that they wanted to learn more about computer 
networking and thought the resources, especially Gopher and Mosaic, were incredible. In the first session,, 
teachers began to see the value of the network to their curricula; but they trucly comprehended the Internet's 
(power, importance, significance) by the second session. Teachers stated that networking was valuable in their 
classrooms, lesson planning, and curricula in the following ways: doing research, sharing information .with 
other schools across the nation and world, ^eating statistical data for debates and persuasive speeches, 
corresponding by e-mail , problem sharing/solving, finding interests of students, and locating career 
information/suggestions. Teacher's comments about the potential benefits of computer networking included: 
excellent research tool, expanding the walls of the media center, more resources, more options, limitless, too 
many to list, new method of instruction, and more efficient use of time. 

Teacher Use of the Network 

Teachers continue to use e-mail and the Internet for research and to share ideas. For specific research topics, 
teachers have searched the Internet by keyword. One social science teacher used 1990 census comparative data to 
gain current, accurate, information about Los Alamos fora student video project. Social science classes studying 
New Mexico have used the National Weather information and satellite images for ongoing reports on New 
Mexico weather. Another teacher asked for lessons or information on civil rights and/or prejudice to use in 
conjunction with a unit on Dr. Martin Luther King. He was amazed by the variety and excellence of the lesson 
plans that he found on the Internet. These are only a few examples; the list grows daily. It is very encouraging 
to teachers to be able to find something and say, "This is exactly what I need!" 

The teachers also use the network to get the "latest breaking news" on topics for use in their classrooms. For 
example, the day after the recent earthquake in Los Angeles, a teacher used the Internet to get the latest breaking 
news as well as general information about earthquakes to use with classes that day. 

Important Note: These are primarily novice users. The initial teachers 1 workshops were held only 
three months before the writing of this paper, 

V. STUDENT TRAINING AND INTERNET USE 

Shortly after the first teacher training workshop, a group of 15 students from the 8th grade Gifted and Talented 
Education Program (GATE) was introduced to the network. Each student was given an e-mail address, password, 
and a Eudora disk. The lead teachers then taught the students how to send e-mail. They began by sending mail 
to each other and then to friends, teachers, and family members, as well as to peers outside the school network. 
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After the initial session, students were invited to use the computers during open periods, and they took full 
advantage of this invitation. During sessions, it was not unusual to hear a student exclaim, "Hey, 1 got a 
message from my dad!" or "I have to leave a message for my mom to tell her that I'm going to Susie's house 
after school" 

News at the school travels about as fast as e-mail on the Internet; so right after our first session with 
students,we had about ten students approach the lead teachers for e-mail addresses and access to the Internet. The 
lead teachers decided to hold another training session for the original 15 GATE students to introduce them to 
Gopher and Mosaic before starting with another group. In that session, the students quickly found shareware 
software, census information, and satellite images using Gopher. They also discovered Dr. Fun and museum 
resources with Mosaic. 

Several new introductory sessions have now been held for other groups of students. Students can sign up for an 
Internet class and get addresses during a "Stretch" period," (two-hour periods, offered once a week, when students 
can choose special activities). The first to sign up get in — the sign-up list for the "Internet Stretch" is always 
lull! 

We also discovered Listserv information and suspected it would be of interest to the students; we were right — 
they took off. One particularly valuable Listserv was KIDLINK, This links kids, ages 10 to 15, through c- 
mail to other kids around the world. By this time, more students had asked for access to the Internet and e-mail 
addresses. We have students sign an agreement about proper use of e-mail along with their application for an e- 
mail account. 

Wc have received mail from kids in many countries including Norway, Germany, and Russia. As David 
Hollman, a member of the first Global Classroom Youth Congress slated, "There's no way I could have opened 
a textbook and gotten the same information I obtained from online dialogue with peers in Israel, Russia and 
China." (Itzkan, 1994). 

After approximately three months 69 students have attended e-mail/Intcmet classes and are using e-mail and 
Internet resources — with classes and individually, In addition to these voluntary sessions, classroom teachers, 
assisted by the ~ad teachers, bring entire classes to the computer lab to use the Internet for specific purposes and 
activities. 



LANL staff provided technical guidance and support for hardware connectivity, network growth, and installation 
and use of network/user software. They helped the middle school staff network their computing facility to the 
outside world with Internet and e-mail, and now they continue to provide consulting support. When a problem 
was unrelated to the network connection itself, e-mail emerged as the most effective method of consulting. 



Clearly, this model of school networking is a stage in an evolutionary process. Lessons learned arc numerous 
and, as shown in the problems section above, the installation of the network in an educational setting with a 
pre-existing computer lab had both advantages and disadvantages. However, now that everything is up and 
running, wc find that keeping up with the usage and application needs of motivated teachers and students is the 
primary challenge. 

As this paper goes to press, the model illustrated here is being used to design an educational networking system 
in another school district. Lessons learned here, especially the keys to success, are being directly applied to this 
new project. The model will be refined as new testbeds yield new insights and information. 

Installation of the technology, however, is just the first step in experiencing the Internet in an educational 
setting. As collaborative learning becomes a major form of instruction, new and ycl unknown Internet learning 
experiences will occur; thus new educational uses for the Internet will emerge. We therefore see our next step to 
be continued evaluation and refinement of this model as wc wait, watch, document, and continue to learn. 



VI. TECHNICAL AND CONSULTING SUPPORT 



VII. SUMMARY 
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Abstract: A study of available authoring packages was made. It was realised that object 
oriented techniques and a grapic user interface (GUI) such as Microsoft Windows would have 
to be used. The reasons for selecting MS Visual Basic as the authoring language is presented. 
In its standard form VB does not contain response analysis algorithms and the sophisticated 
text handling needed for CBI. However, it's open structure allows custom controls and DLL's 
to be incorporated. The specifications of these utilities and their programming in C/C++ will 
be discussed. The aim is to provide an efficient authoring tool for the lesson designer and 
programmer. The relatively low price of VB and the value of royalty free distribution of 
courseware written in VB is pointed out. 



Introduction 



About two years ago I saw Microsoft Visual Basic for the first time and decided that 1 was 
looking at the future of computer programming — period 

Suddenly, here was a tool that allowed me to write applications in hours that literally used 
to take two weeks. What's more, it was powerful and extensible enough for use in serious 
professional applications. 

~~ — "~ ~ Dan Appleman. 1993.Visual Basic Programmer's guide to the Windows API. 

(PC Maga2ine, ZD Press) 



This quote from Dan Appleman's book portrays the same excitement that we experienced on our first exposure 
to Microsoft's new programming language for Windows, Visual Basic. We, however, were looking at VB with 
completely different eyes. We saw the potential of turning VB into a powerful authoring tool for producing 
sophisticated CBE courseware efficiently and fast. 

Why were we looking at authoring systems? The Gold Fields Computer Centre for Education at the 
University of Pretoria has a dual network for delivering courseware to eighty student terminals. The first one is 
the mainframe-based CDCNET on which CYBIS (successor of PLATO) is run. The second network is an up-to- 
date ethernet/Novell LAN driven by a 486 file server. The same IBM compatible PC's serve as terminals for 
both systems. For a number of years CYBIS was the backbone of activities at the centre. High maintenance 
costs and an increasingly unreliable CYBER have precipitated the decision to phase out the mainframe while 
developing software for the LAN. Thus our interest in finding a suitable authoring package. We wanted high 
productivity with state-of-the-art quality and sophistication. At the same time we want to promote the 
widespread and large scale use of computer assisted learning in our region. We decided that a graphics user 
interface (GUI) and object oriented techniques were desirable, with Microsoft Windows being the obvious 
choice. The Windows packages we investigated were either prohibitively expensive, or not strong enough in 
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certain essential areas such as response analysis, or plagued with annoying royalty and licensing restrictions. 
Visual Basic, being a programming language not specifically designed for the production of courseware, was 
also lacking in two important ways: response analysis and a symbol editor for scientific applications. However, 
it had none of the other disadvantages. It's visual interface was very easy to use and it's open structure was 
particularly attractive. Apart from a full set of Basic language functions, most of the functions from the 
Windows API can be incorporated. In addition, the full power of the C/C++ language can be used to program 
add-on procedures, functions and custom controls. We realised that with some late night programming we could 
fill in the gaps to turn VB into a system ihat would meet our requirements handsomely. 



To learn is to create 



The ultimate objective of any computer-based instruction is to create an effective and meaningful learning 
environment. Educationists have been studying the learning process for centuries. In our view one of the key 
factors in effective learning is creativity. Creativity is an indispensable part of learning. Furthermore, creativity 
and learning are very personal concepts. Nobody can learn something on someone else's behalf. Learning is the 
personal creation of knowledge and teaching is facilitating this creative process. In computer-based instruction 
creativity (learning) is promoted by establishing continuous interactbn with the learner at the terminal. The 
normal way to do this is to pose a question to which the learner must create an answer. The learner's attempt 
must then be analysed and judged, not only for correctness, but more importantly in the case of a wrong 
response, for the flaw in the reasoning. Meaningful feedback should then be given to the learner as well as the 
opportunity to try again. The purpose of these guidance questions is not to test the knowledge of the learner, but 
rather to focus the mind and to guide him or her to gain new knowledge. Often a wrong response provides the 
perfect opportunity to teach. The full facility of the computer system should be available when compiling the 
feedback. 1 t feedback could be another question to probe deeper and perhaps find the student's misconception. 
It could contain all the multimedia components and could branch to another section of the lesson or even to a 
different lesson. Programming these feedback loops can be very tedious if the authoring package is not geared to 
help the lesson designer. It is heart-breaking to see how often the feedback, where it matters most, is treated 
superficially, and how often the correct answer is simply given after one or two wrong attempts, before the 
learner had a chance to discover (create) the knowledge. Some authoring systems allow only a correct/incorrect 
feedback to a student's response, which is hopelessly inadequate. 

Another consequence of the reasoning given above is that we try to avoid multiple choice type 
questions wherever practically possible. Pick & choose type interaction leaves little scope for creativity. As a 
result learning is not very effective and retention is low. True/false type questions are even worse. Many of the 
features of the authoring shell which is described in the following paragraphs are intended to simplify the proper 
implementation of the response analysis - feedback loop. 



The Visual Basic Scene 



Figure 1 is a screen dump of the VB design time screen. It is a 800X600 SVGA screen, where the central form is 
a 640x480 area which will become the student screen at run time. (On the illustration the image has been 
reduced to a practical size for printing). It is not the purpose of this paper to explain all the standard features of 
VB. A description of these features can be found in the manuals, and many people are familiar with VB anyway. 
We would, however like to point out tlie features which are of particular importance from the lesson designer's 
point of view. 

On the left hand side of the screen the control box is shown. It contains the standard controls as well as 
any number of custom controls which could be added when needed. The two last controls, the foot print and 
ANS, are custom controls created by us. They form the core of our VB authoring system and will be discussed 
in some detail. Hundreds of other custom controls are continuously released by independent developers world 
wide. Apart from the possibility of creating one's own, the lesson designer has an incredibly rich selection of 
controls at his disposal. 

Controls are dragged onto the form where they become objects. Each object has a numbev of pre- 
defined properties. The values of these properties can be changed on the properties window. Basic language 
code can be attached to each object. This code is activated by certain events such as mouse button clicks and key 
presses. A lesson or application would consist of a number of forms, each containing a variety of objects. These 
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FIGURE 1 

objects can be independent or linked in many different ways. The object oriented nature of VB programs gives 
tStSl to the lesson designer to indiv.dualise navigation through a lesson, w.th minimum ^ode and very 
fittle programming effort. Educationally this is an important feature and m harmony with our demand of 
maximum creativity ^ ^ ^ ^ ^ rf ^ properties ^ for 

the standard VB text box and deal with border styles, colours and fonts. The additional ones, however, give us 
the Soring fea^s. If the Question property is FALSE the object is a text box with enhanced border options, 
but some other P'^^ 

property is Aen used to cLifyThe type of response expected. This information is used to select the appropnate 

response analysis algorithm. 

At Dresentthe following types of responses are foreseen: 

0. Number ' A numeric response with awareness of the correct aumber of significant figures, but without 

1 . NumberUnit: The same as Number but with units. Automatic adjustment for different correct units is 
incorporated. 

9 Text* Judced for key words with spelling tolerance. 

I MathExpfession: Symbolic mathematical expression. Automatic allowance for different correct forms of 

4. MaSutlon; Two mathematical expressions separated by an equal sign. Alternative correct grouping of 
symbols allowed. 

5 ChemExpression: Chemical expression. Correct alternates allowed. 

6 ChemEquation: Two chemical expressions separated by an equal s.gn or arrow. 

7 Diacram: A diagram created by dragging and dropping pieces. 

s'. MultipleChoice: One or more correct alternatives and a number of distractors. 
9. Match: Matching of two columns of information. 
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Not all of these types will be implemented mediately 
but the response analysis algorithm library will be ontinuously 
developed and enhanced. New types will also be added to the 
list when needed. One challenge we intend to take up in future 
is to provide response analysis for music notation. 

TheFeedback property can be set to FALSE. This will 
be the case when the question is part of a test where the 
response is judged and graded, but no feedback given. Online 
tests and examinations could be set very easily and quickly. 

The Restart property sets a restart flag. Each ANS 
object will also be a restart point in the lesson when Restart is 
set to TRUE. The learner will return to the most recent ANS 
object with a TRUE Restart property when he restarts a lesson 
that was interrupted. 

The TagNo property is an identification tag which is 
automatically allocated to each ANS object by the system 
when it is created. This tag is used by the FOOTPRINT control 
to keep track of the ANS objects throughout the lesson. If the 
designer does a right button click on the foot print a list of all 
the existing ANS objects is displayed in a pop-up window. The 
list indicates the settings of the Question and Restart properties 
as well as the first few word of the text in the ANS box. This 
feature is intended to help the designer to have an overview of * 
the lesson and to find the reference for branching to any ANS 
object quickly. The FOOTPRINT only appears on the design 
screen and has no impact on the screen at run time. 

The Response Dialogue Window 

Each question normally has a correct response (or a few 
alternative correct ones),but there are many ways to answer a 
question incorrectly. It is impossible to treat all these wrong 
options in a program of practical proportions. The experienced 
educator who knows his subject well, also knows which wrong 
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answers occur frequently, and what flaws in the reasoning cause them. It is therefore possible to anti Z a 

n 2*f TT 8 anSWeRi "*? Pr ° Vide feCdback t0 C ° mct them - We decided > 35 a s^inS P°int, to aSow for 
ten different Responses to each quest.on, one correct one and nine wrong ones. The number of ten is arbitrary 
and only one constant ,n the source code needs to be changed should a different number of responses be 
(0 lT TSe 11 15 not necessai ? t0 use a " ten for ^ery question). To strengthen the feedback in 
accordance w.th the .mportance we attach to it, we make provision for an optional second feedback to every 
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FIGURE 4 

wrong response. The diagram in Figure 3 shows this schematically. When the Question property is TRUE a right 
button click on the ANS object activates the response dialogue pop-up window as shown in Figure 4. This 
feature is a design facility only and is not active at run time. The designer can click on the NEXT and PREV 
buttons to page through the ten responses. Apart from the buttons at the bottom the dialogue window is divided 
into three sections. 

The top section contains the anticipated RESPONSE and information associated with it. The co- 
ordinates of the rectangle where the response will be entered is displayed on the right. The rectangle is 
positioned visually on the screen, but the designer can override this by editing the numbers, especially when 
small changes are needed. The buttons below the text box are used for special characters by the scientific editor 
(see par. 5). If the question is to be graded, a mark can be entered into the "Mark" box. 

The second section contains the FEEDBACK. When the Feedback property is set to FALSE this part of 
the window does not appear. Two text boxes are provided for the first and optional second feedbacks. Normally 
the feedback stays on the screen when the student tries again. If the "Erase" buttons are switched to "on" the 
feedback is erased. If the same wrong response is given a second time the second feedback is used. It can either 
replace the first feedback or it can be appended to the first feedback according to the setting of the "Replace" 
and "Add" buttons. The response and feedback data are stored as invisible properties of the ANS object. 
Sometimes tho feedback will be the Same as the feedback to a previous question. In that case the designer can 
fetch such feedback by using the "Fetch" buttons. This saves memory and retyping of previously entered text. 

As stated earlier we feel that the full facility of the system should be available to be used in the 
feedback to the student. The buttons in the lower left corner are meant to add different colours, a bitmap and 
multimedia features to the feedback. These things are of course standard features of Windows and VB, but it 
should be understood that these buttons are used to attach a bitmap, let's say, to a feedback to a specific response 
to a specific question. 
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The third section controls the CONTINUATION after the question has been answered. The student will 
either try aga.n or continue to the next object in the lesson. Sometimes the designer might feel that ordinary 
feedback is not enough to resolve the learner's problem and that he (the learner) should work through another 
section of the lesson before his next attempt at answering the question. The branch button activates this option 
By clicking the "Get Branch" button the FOOTPRINT list window is displayed. The designer then selects an ' 
object from the list and enters the reference into the box by clicking "GetBranch" again. The "Return Here" 
button determines whether the learner returns to this point in the lesson after the branch. The branching facility 
is particularly useful and important when a secondary question is needed as feedback. 

The Scientific Editor 

As mentioned earlier, very little emphasis is placed on typed answers in current CBI authoring packages Of 
those that do accept typed answers, even less thought is given to an author wishing to have the student input an 
answer using mathematical symbols and/or superscripts and subscripts. The most obvious reason for this is 
firstly, the difficulty in analysing the response, and secondly, the student normally has to learn a complicated 
sequence of keystrokes to produce one of these characters. We have found that this, more than anything else 
frustrates the student, and as such detracts him from the learning process. We decided that all aspects of the ' 
input mechanism should be completely visible to the student, so that if he can operate the mouse and the 
keyboard, he has all the tools required to answer a question. Special character selection is then as simple as using 
the mouse to choose a character from the "softkeys" such as those shown in the diagram below. 

The diagram below is from the lesson developer's editor, which is a superset of the student's editor. The author 
has the extra features shown on the menu bar which allows him normal editing facilities and the ability to set 
special characters on buttons which the student may or may not require. Of course if this is to be a "text box" 




FIGURE 5 

then all that will appear in the window on the student's screen will be the "texf.The scientific editor has been 
written in C++ to make full use of dynamically allocated objects to store items such as the buttons required to 
answer a question, as well as the answer itself. The parser then uses this same information in analysing the 
answer. The editor and the parser are in a standalone DLL, and have a series of C functions to allow other 
windows systems to communicate with them. 



The Manager 

VB3.0 alsc features a database control with a native MS ACCESS format. However, it can connect to several 
popular database formats such as dBase and Novell's Btrieve. Because it is internal to VB, it is very easy to use 
it as the database to control the Lesson Delivery System (LDS). This database needs to contain, for each student 
details of the current curriculum, the current lesson from the curriculum, restart markers and statistics such as 
lesson usage. The LDS can then load the required lesson, and update the database continuously on receiving 
information from the lesson. The manager is the final phase of this project. 
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Conclusion 

Although all the features mentioned in this paper are not implemented yet, we feel that Visual Basic and our 
authoring custom controls will soon provide us with the authoring system we need. As mentioned earlier an 
important advantage of using VB as an authoring package is cost effectiveness. The standard package costs a 
fraction of the price of established authoring systems. Furthermore, al! compiled programs written in VB can be 
distributed free of any royalties and licensing restrictions. 
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Abstract: Although widely accepted as fundamental to the instructional design 
process, learning hierarchies (Gagne, Briggs & Wager, 1988) do not fully specify the 
presentation sequence of a set of instructional objectives. Memory load theory 
(Nesbit & Hunka, 1987) more thoroughly exploits the structure of learning 
hierarchies by proposing that the most effective instructional sequences minimize the 
distance over which prerequisite links are extended. The present study investigates the 
extent to which textbook authors order sections within chapters to conform to this 
principle. Twelve chapters from math and science textbooks were analyzed to obtain 
underlying learning hierarchies. The hierarchies were compared with the actual 
sequences chosen by the textbook authors. There was strong evidence that authors 
preferred sequences characterized by high prerequisite coherence. It appears that 
authors do not consciously strive to minimize memory load, an optimization 
problem that is generally intractable, but rather follow a depth-first satisfying 
strategy. Implications for the design of computer-based instruction are briefly 
discussed 



The sequencing of instructional presentations is a problem of long-standing interest to 
instructional psychologists, and one for which a variety of prescriptive theories have been 
proposed (Posner & Strike, 1976; Patten, Chao & Reigeluth, 1986). Prominent among these is 
the highly influential formulation originated by Gagne (Gagne & Paradise, 1961; Gagne, 1962) 
and stated most recently in Principles of Instructional Design (Gagne, Briggs & Wager, 1988). 
Though rooted in the transfer of learning psychology of the 1950s, Gagne f s theory has proven 
remarkably portable across the collegia! boundaries that separate specialists in training, 
instructional technology, education, and psychology. It has remained a mainstay in handbooks 
and textbooks dealing with instructional design (e.g., Jonassen, Hannum & Tessmer, 1989), 
yet has been taken up, often in variant forms, by researchers applying cognitive science to the 
development of intelligent tutoring systems (Lesgold, 1988). 

To design instruction for intellectual skills, or procedural knowledge, Gagne advocates a 
form of task analysis which recursively decomposes a goal skill into sub-skills linked by the 
prerequisite relation. The resulting structure, called a learning hierarchy, has three vital 
functions in the planning and execution of instruction: (1) it defines a partitioning of the content 
into discrete instructional objectives, (2) it specifies restrictions on the order in which 
instructional units covering the objectives may be presented to the learner, and (3) when 
learning fails it points to prior objectives that are candidates for review. 

The size and content of each objective in a learning hierarchy is primarily determined by the 
following criterion: If all prerequisites are in place and minimal instruction is supplied, the 
objective should be sufficiently simple such that the learner will be able to immediately achieve 
a successful, though not necessarily masterful, performance of the skill addressed by the 
objective. In some domains a hierarchical analysis can enable the supplied instruction to be 
reduced to only a series of guiding questions or problems posed to the learner. The aim of a 
hierarchical analysis, then, is that instruction be so formed and ordered that the learner be ready 
to rapidly assimilate the target skill when supplied with minimal new information. Because this 

76 



ERIC 



81 



Nesbit, J. C. (1994). Prerequisite Coherence in Instructional Presentations . In M. Orey (Ed.), Proceedings of ike Thirty-fifth ADOS 
Conference (pp. 76-83). Nashville, Tennessee: Omni Press. 

sort of analysis defines each objective under the assumption that all its prerequisites will be 
available to the learner, it is crucially important that when the moment for transfer arrives 
prerequisite knowledge not only have been attained but also retained 



Memory for Prerequisites 

For facilitating access to attained prerequisite knowledge, there are three mutually 
compatible types of strategies available to an instructional system. 

(1) At time t 1 , when prerequisite knowledge is acquired, deploy instruction which ensures a 
durable and sccussible memory trace: Provide practice, mnemonics, schema induction, and 
so on. 

(2) At time t 2 , the moment of transfer, deploy instruction which stimulates effective retrieval 
of prerequisites: Provide cues to guide memory search, activation of relevant schemata, 
analogies that illustrate the relation between prerequisite and current objectives, and so on. 

(3) Minimize the interval between t 1 and t 2 . 

The first two strategies are explicitly recognized by Gagne: 

To be most effective for new learning, prerequisite skills must be 
thoroughly learned, that is, learned to mastery. Presumably, this degree of 
learning makes the prerequisite skills easier to recall and, therefore, more 
readily accessible for new learning. Another condition that affects the ease 
of retrieval is the number of cues available to the memory search process. 
Aids to memory search are provided by the cues of a schema. Accordingly, 
embedding the prerequisite skill (or skills) within the organized framework 
of a schema may be expected to have a beneficial effect on instruction. 
(Gagne, Briggs & Wager, 1988, p. 109-110) 
But there are costs accompanying strategies that attempt to strengthen encoding or stimulate 
recall. Instruction on already comprehended knowledge takes additional time, and, especially if 
in the form of repetitive practice, can claim a toll on learner motivation. Moreover, there are 
instances where heavy mastery of part of a skill can interfere with later acquisition of the whole 
skill (Singley & Anderson, 1989, p. 114), as demonstrated in famous early studies on the 
Einstellung effect (Luchins, 1942). 

Processing of retrieval cues must also be counted as a cost in cases where the prerequisite 
knowledge is already activated. Parallels can be drawn with the use of anaphoric reference in 
discourse: The clearest constructions delete explicit cues to information that is already 
established in the working memory of the listener. Similarly, in designing instruction, the use 
of elaborate prerequisite retrieval cues should be restricted to only those transfer points at 
which they are really needed. 

Although largely neglected in the literature on learning hierarchies, the third and most 
passive of the three strategies has the advantage of requiring no additional instruction; and thus 
offers the greatest leverage against the tyranny of the total time law. If minimization of the 
prerequisite retention interval is given priority, then the two strategies which more actively 
promote access to prerequisites can be deployed as required 

There is evidence that the prescription to minimize prerequisite retention intervals is not 
simply an unarticulated assumption of learning hierarchy theory. While expressing a concern 
for retention of prerequisite knowledge, Lesgold (1988) could find no compelling reason to 
prefer depth-first traversal of the hierarchy over breadth-first traversal, concluding that n there 
may be individual differences in aptitude or preference for these two approaches". From 
memory load theory we can arrive at the contrasting position that a depth-first traversal is more 
effective for all learners because it generates shorter retention intervals. 
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Prerequisite Coherence 

Factorial explosion ensures that a sei of only a dozen instructional objectives has many 
millions of possible orderings. To specify a learning hierarchy is to remove from consideration 
a large fraction of these sequences; yet a typical hierarchy can leave millions of sequences as 
valid possibilities for presentation to the learne*. Memory load theory begins with the question: 
Can all sequences in such a large set be equally effective? 

Nesbit and Hunka (1987) advanced the view that instruction is most effective when 
objectives are presented in sequences minimizing the interval over which prerequisite links are 
stretched. Here it is further argued that teachers, instructors and authors have an untutored 
tendency, when informed of the prerequisite relations among objectives, to choose sequences 
exhibiting this sort of coherence. 

For the three hierarchies shown in Figure 1, prerequisite coherence can be illustrated as a 
property of those sequences which contain the segments VW and XY. For example, the 
sequence UVWXY is a coherent traversal of hierarchy 1. 




Figure 1. Three hypothetical learning hierarchies. 

A simple way to quantitatively capture the memory load imposed by a sequence of 
objectives is to first assign a duration (i.e., an estimated learning time) to each objective, then 
sum the durations over which each link is stretched, and sum over all links. Figure 2 shows 
that, if a duration of 1 time unit is assigned to all objectives, the coherent sequence UVWXY 
for hierarchy 1 would evaluate to a summed duration of 2 time units, while the less coherent 
sequence UVXWY would evaluate to 3 time units. To allow comparisons across hierarchies, 
the summed duration for a sequence is normalized by obtaining its reverse-order percentile rank 
among the population of sequences permitted by its hierarchy. A sequence uniquely possessing 
the least summed duration of all sequences permitted by a hierarchy would thus have a 
prerequisite coherence approximating 100, while the coherence of a sequence uniquely 
possessing the greatest summed duration would evaluate to 0. 

The two examples in Figure 2 highlight major facets of the prerequisite coherence concept. 
The coherent sequence (UVWXY) results from a depth-first traversal of the hierarchy, while 
the incoherent sequence (UVXWY) results from a breadth-first traversal. More generally, 
depth-first traversal does not guarantee maximum coherence but does seem to work as a 
serviceable heuristic. As further demonstrated by Figure 2, breadth-first traversal tends to 
uniformly distribute retention intervals over prerequisite links, while depth-first traversal tends 
to distribute unevenly by allocating most of the summed retention duration to a small subset of 
the links. Again, depth-first traversal should be preferred because (1) it allows better prediction 
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of prerequisite knowledge likely to be forgotten and (2) due to the non-linearity of forgetting, it 
imposes a lower actual memory load for the same summed duration. For the coherent sequence 
UVWXY we can readily predict that the prerequisite link from U to X is the most nicely to need 
additional instructional support, perhaps in the form of reviewing or cueing U. At the same 
time, the non-linearity of forgetting curves tells us that dividing a retention interval between 
two items results in greater total information loss than allocating the entire interval to a single 
item. 



sequence of objectives 



0 0 




sum of retention intervals 
0+0+0+2=2 




9 

ERIC 



0+1+1+1=3 



1 

Figure 2. Estimating memory load effects by summing retention intervals. 

To this point in the presentation, a simple definition of prerequisite coherence based on 
summation of retention intervals has been elaborated. A relative measure of coherence has been 
obtained by converting a summed duration into a percentile, such that the most coherent 
sequence for a hierarchy produces a coherence measure of 100. But without data we cannot be 
assured that this definition gives the most accurate measure of coherence. For the time being 
we need to work with a set of measures that can be compared and tested as predictors of learner 
performance and instructor preference. Theory-based variations on the core definition 
constitute an extended family of coherence measures, with each member representing a trade- 
off between definitional simplicity and theoretical validity. For instance, a power function ot 
the retention interval can be introduced to better account for the fact that retention falls ott non- 
linearly with time. Note that a linear coherence measure will not distinguish between deptn-tirst 
and breadth-first traversals of hierarchy 3 of Figure 1, but that introducing a more natural non- 
linear function of the retention interval, such as square root, will result in a higher coherence 
rating for depth-first traversals. Another variation used here foUows the principle tnat the 
retention interval should be calculated from the most recently attained objective having the 
prerequisite as a component skill, rather than from the prerequisite itself. Which variations best 
serve the instructional designer is a question to be addressed by empirical work. 

For the purposes of the present study, a series of four increasingly refined prerequisite 
coherence measures were constructed: D w , D m , D°\ D (4) . In order to define these 
measures, a few symbols and terms are now introduced. A learning hierarchy, and every 
sequence generated from it, contains m links or arcs, each connecting a prerequisite to a target 
objective. Each arc i of a sequence stretches across q, objectives which intervene between its 
prerequisite and target objectives. Each arc of a sequence also has a single sufficient prior 
objective, which is the last (rightmost) objective before (left of) the target objective that either 
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has the prerequisite as a component skill or is itself the prerequisite. For example, in the 
sequence UVWXYZ from hierarchy 3 of Figure 1, X has U as a prerequisite and W as a 
sufficient prior objective. Four estimates of memory load effects are now defined: 

A count of intervening objectives over which 
prerequisites must be retained. 

A sum of retention intervals over which 
prerequisites must be retained; is the 

estimated learning time of the jth objective 
intervening between the prerequisite and target 
objectives of the ith arc. 

A sum of memory loads estimated as a non- 
linear function of retention interval. 



A sum of memory loads estimated from 
sufficient prior objectives; s u is the estimated 

learning time of the jth objective intervening 
between the sufficient prior objective and target 
objective of the ith arc. 

Each prerequisite coherence measure D is then a reverse-order percentile indicating the 
position of the target sequence in a distribution of estimated memory load effects d of all 
sequences generated from the same hierarchy. 

The purpose of this study is to test the hypothesis that instructional texts show prerequisite 
coherence. Evidence for or against coherence in texts does not constitute a direct rejection or 
acceptance of the more central claim that prerequisite coherence facilitates learning; but it would 
contribute to or detract from the construct validity of memory load theory* If instructional 
design for computer-based instruction is construed as an enterprise of automating and 
systematizing the processes by which instructors arrange conditions for learning, then the 
finding that texts exhibit high prerequisite coherence would reveal a useful criterion for 
automated design. A separate experiment testing the central claim that prerequisite coherence 
actually facilitates learning is now nearing completion. 



Method 

Textbook chapters were analyzed to obtain underlying learning hierarchies. With these 
hierarchies in hand, prerequisite coherence values were calculated for the actual sequences used 
by the textbook authors. 

As a preliminary step to this procedure it was necessary to develop operational definitions 
for the concepts of instructional objective, sequence, prerequisite, and learning time that play 
major roles in the construct of prerequisite coherence. It was decided to let a textbook chapter 
represent a single sequence of objectives, and to let chapter sections as denoted by the textbook 
author stand for instructional objectives. A chapter section was only considered to be a 
prerequisite of a later section if understanding or mastery of the later one clearly required 
understanding or mastery of the earlier one. Reference to terminology introduced earlier did not 
alone constitute a prerequisite relation if the terminology could have been briefly defined in the 
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later section. Learning time for a chapter section was estimated by a count of sentences — the 
large sample of text forbade use of more precise estimates obtainable by propositional analysis. 

Twelve chapters from mathematics and science textbooks were selected In comparison 
with other academic subjects such as literature, prerequisite relations are more readily 
perceptible in mathematics and science because they include a larger proportion of procedural 
knowledge. Other selection criteria were: 

• The hierarchy derived from the chapter should provide a sufficient basis to test a 
preference for prerequisite coherence. Occasionally hierarchies were found which 
completely determined the presentation sequence. In a few other cases, all possible 
traversals of the hierarchy generated the same coherence value. These types of 
hierarchies, which were rejected, tended to occur only when the number of chapter 
sections was very small. 

• The number of sections per chapter should not be too large. Several chapters with more 
than 10 sections were analyzed and rejected because the numbers of sequences 
permitted by their hierarchies were too large to be counted by my computer in a 
reasonable length of time. 

• The analysis of prerequisite relations in the chapter should permit an unequivocal 
determination of its hierarchical structure. Several chapters were rejected on these 
grounds. 

It is important to note that chapters were never rejected after calculation of coherence values, 
except in those few cases in which all traversals generated the same coherence value. 



Table 1. Prerequisite coherence of 12 textbook chapters. 

Percentile of actual sequence 



Text 




Chapter 


Sections 


Sequences 




D<2) 


DP) 


DW 


Dorin et al. (1989) 
high school chemistry 


1 


chapter 9 

chemical equations 


13 


40734 


75 


99 


99 


100 




2 


chapter 19 
acids, bases, salts 


12 


20228 


73 


42 


79 


98 


Duncan (1982) 
high school physics 


3 


chapter 9 
fluids at rest 


13 


180469 


100 


100 


99 


69 


Fisher (1977) 

college discrete mathematics 


4 


chapter 3 
graph theory 


9 


38 


100 


100 


100 


97 


KaILshetal.(1980) 
college logic 


5 


chapter 2 

propositional logic 


9 


35 


94 


94 


99 


86 


MoycrA Bishop (1986) 
high school general science 


6 


chapter 10 
mechanics 


8 


576 


98 


97 


98 


98 


Smoot et al. (1987) 
high school chemistry 


7 


chapter 12 
chemical bonding 


10 


900 


94 


89 


93 


83 




8 


chapter 20 
thermodynamics 


10 


6 






100 


80 


Stanat & McAllister (1977) 
college discrete mathematics 


9 


chapter 3 
binary relations 


7 


12 


95 


95 


91 


77 


Strcctcr & Alexander (1989) 
arithmetic for adult learners 


10 


chapter 5 
fractions 


6 


20 


97 


100 


100 


92 


Zumdahl(1989) 
college chemistry 


11 


chapter 3 
stoichiometry 


9 


81 


100 


100 


100 


99 




12 


chapter 4 

chemical reactions 


12 


42690 


94 


100 


100 


100 
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Results 

Tabic 1 shows the four prerequisite coherence values for each of the 12 chapters. The table 
also indicates the general content of the textbook, the number and content of the chapter, the 
number of sections in the chapter, and the number of sequences permitted by the learning 
hierarchy. Two coherence values arc omitted for entry 8 because measures DW and D< 2 > 
assigned the same coherence value to all 6 sequences permitted by the hierarchy. 

The only coherence value falling below the median is measure D< 2 > of entry 2. Other 
measures produced high coherence values for that chapter. Most of the chapters exhibited 
coherence close to the maximum value. 

If textbook authors were indifferent to prerequisite coherence we would expect coherence 
values to fall above and below 50 with equal likelihood (recall that our measure is a percentile). 
Chi-Square analysis permits a rejection of the indifference hypothesis with respect to all four 
measures: 

DO) X2^n.0,p<.001 
D») %2= 7Ap<.01 

DP) and DW %2 =12.0, p < .001 

If only one chapter is allowed per textbook in the interests of independent sampling, then 
significance is still obtained for all measures (p < .05). 

Evidence for differences in reliability among the four measures of prerequisite coherence is 
considerably weaker, There are few cases where the measures show marked disagreement; and- 
no consistent pattern of discrepancy is apparent. 



Discussion 

The results indicated a tendency among textbook authors to deploy instructional sequences 
with prerequisite coherence far greater than that expected from random traversal of the 
underlying prerequisite hierarchy. The question remains whether this tendency figures 
prominently in the instructional planning of authors, or whether it emerges as an 
inconsequential side effect of some separate process. Evidence that learning is facilitated by 
highly coherent sequences would support both the primary conclusion that prerequisite 
coherence is a fundamental property of effective instruction, and the secondary conclusion that 
it receives the direct attention of textbook authors. 

Finding the most coherent sequence for a sizable hierarchy is generally an intractable search 
problem for even very powerful computers. This being so, how are authors able to generate 
highly coherent sequences of chapter sections? Further analysis of the chapters listed in Table 
1 indicated that authors rarely deviated from a depth-first traversal of the learning hierarchy. 
While falling short of perfection, depth-first traversal can routinely generate highly coherent 
sequences. 

In this context, depth-first traversal means that after completing an objective i the author 
proceeds with another objective j such that i is an immediate prerequisite of j. The depth-first 
strategy does not always fully determine the sequence because an objective may have more than 
one prerequisite. A major practical advantage of depth-first traversal is that it requires only local 
knowledge of the hierarchy. In other words, the author need not be aware of the entire 
hierarchy as she chooses the next objective in the sequence — only prerequisite links 
emanating from recently presented objectives are of concern. 

Prerequisite coherence is readily applicable to a range of computer-based instructional 
systems. It is a simple matter to devise hierarchy searching programs that generate sequences 
with higher prerequisite coherence than those typically chosen by authors. As long as a 
learning hierarchy is available, such programs can suggest one or more sequences to a course 
author, a student, or an automated planning module. The program searching for coherence can 
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continue its work while the student progresses through the hierarchy, modifying its 
recommendations according to the learning times and sequence actually experienced by the 

* tUC The most fundamental conclusion drawn by MacMillan, Emme and Berkowitz (1988, p. 
244) in their analysis of problems faced by intelligent instructional planners was that 
"instructional planning is an underspecified process in a large planning space". In this regard, 
the principle of prerequisite coherence can be wielded by planners as a widely applicable 
criterion for constraining search through the instructional planning space. 
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Delivering Multimedia Lectures via Fiber Optic 
Two- Way Interactive Video 
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Abstract: Fiber optic two-way interactive video (ITV) courses for rural Western Kansas were planned 
as technician-supported endeavors. Multimedia, videotape clips and special effects would be 
generated by a technician. This plan, however, would have disrupted course development cycles for 
some faculty. The primary author had used multimedia in traditional classrooms; how could this 
technology be brought into the ITV classroom? The solution included merging existing technologies 
of multimedia and ITV. Differences between technology-driven and needs-driven planning are 
discussed. The solution required partnerships; departments that supported or augmented merging 
included: Nursing, Computer Information Systems & Quantitative Methods, Information Networking 
& Telecommunications, the Computing Center, and Forsyth Library, The paper describes the 
partnerships, how the technologies were merged, and how the problem was solved. A short videotape 
"Longitudinal Study: ITV-ship " shows segments of electronic classroom instruction from graduate 
nursing courses, covering a one-year period. Discussion includes how to develop multimedia 
presentations that will also work for future ITV. 

Fort Hays State University (FHSU) completed an electronic classroom in 1992 for two-way fiber optic 
interactive video (ITV) distance learning. The impetus for this paper came when the primaiy author prepared for 
ITV deliveiy to nurses in rural western Kansas. She had been using multimedia lectures in traditional classrooms. 
How could this type of interactivity be brought into the electronic classroom and how would it need to change for 
ITV? During the first ITV semester at FHSU, the primary author used the system for an on-campus course in 
order to learn to use the system and optimize course planning for the following semester. Support resources 
naturally concentrated on courses being delivered to distance sites; therefore, minimal support was available to 
simply learn the ITV system. Partnerships with The Computer Information Systems & Quantitative Methods 
(second author) and the Information Networking & Telecommunications (third author) provided significant 
support (learn-as-you go) for the successes enjoyed the following semester when the primaiy author's delivery 
moved off-campus. 

The paper describes fiber optic technology, the history of ITV at FHSU, the partnerships that developed, and 
transformation stages. Discussion includes how the primary author adjusted to ITV delivery and how to plan for 
future ITV courses. 

Fiber Optic Technology 

There have been numerous transmission media used throughout the modern age; the most conventional arc 
metallic twisted-pair, coaxial cable and wave guides. When evaluating these systems performance standards come 
to mind: how much information can pass per second, and the distance information can go before rcamplification is 
required. The major limitation of traditional analog systems has been limited bandwidth; and information must be 
repeated or reamplified every ten miles. For long distance communication, copper-based systems and microwave 
technology were the only available transmission media until the gradual implementation of fiber optics in the 
1980s. 

Corning Glass Works developed the first feasible fiber optic strands in 1970. These fiber strands had 
attenuation values (ener^ decay) of less than 20 dB/km. This was improved to 4 dB/km by 1972. Current fiber 
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optic manufacturing techniques achieved a value of less than 0.2dB/km. Over one million kilometers of single 
mode fiber had been installed by 1987. 

Optical photonics uses a fiber optic strand that is composed of two simple and basic parts: a core material 
and cladding made of an optically transparent material (such as fused silica glass) and a coating. The strand of 
fiber optic cable is made of silica glass and is between 0.0002 inches and 0.004 inches in diameter. This is roughly 
the same thickness of the human hair" (Jacobi, 1993). The core is the center of the strand and is used to guide the 

,l8ht The core is evaluated by determining the relationship of the core to the wavelength of the signal. It is 
specified by either being smaf er or larger than the signal's sine wave. If the signal's frequency is the same as the 
core then it is referred to as single mode. Cable with a larger core is called multi-mode and has exceptional 
bandwidth When compared, copper wire that can carry 24 voice channels is greatly inferior to fiber optic's 15,000 
voice channels per strand. It is anticipated that channels will be increased to a half-million in the future. In 
addition, fiber can also carry over 160 video channels compared to coaxial cable's 78. More advantages of the fiber 
cable are its easy maintenance, increased service life and resistance to stray radio frequency signals when 
compared to metal-based media. The partnerships necessary to bring fiber to FHSU were numerous and diverse. 

History of ITV at FHSU 

Fort Hays State University is a small regional institution in western Kansas with a 44-county services area. 
The annual enrollment is just under 6,000 students. FHSU is the only four-year liberal arts college in the western 
half of the state (50,000 square-mile service area). The problem for distance learning delivery is compounded by 
the fact that the service area represents almost half of the land mass of Kansas but less than one-third of its total 

population. , . .. .. 

Many of the small rural unified school districts in the center of these communities were finding it 
increasingly difficult to maintain the quality of elementary and secondary instruction. The Kansas Legislature and 
the Department of Education have been in the process of raising standards for secondary schools, to provide a 
quality-based education and insure that the proper courses for college entrance requirements are taught. These 
competencies range from foreign language to art and science. In addition, many schools faced the harsh reality of 
consolidation with other rural districts. The local esprit de corp is influenced by the success or failure of local high 
school sports teams as well as the academic achievements of the sons, daughters and friends attending the schools. 

As a way to insure the longevity of both the rural school and its community, the secondary schools in the 
district decided to pool resources. This would provide for the purchases and installation of a technological 
infrastructure that would allow one teacher to concurrently teach for more than one school at a time. Driving from 
site to site was not practical; the distance between schools could easily exceed 60 miles. 

Public and private partnerships developed among the various sihool districts and the local independent 
telephone exchanges. Community cable systems provided a unified front with a common goal. The goal was to 
provide a secure environment for learning, a sharing of resources, and the development of a diversified curriculum 
while preserving the rural community and its schools. The first fiber optic cluster was activated in the fall 
semester of 1990 Since then, three new clusters became operational to make a total of seven. The results have 
been favorable, with courses in foreign language, biology and mathematics being taught concurrently from one 
school to a combination of three others. 

At the time when much of the planning for the development of Unified School District (USD) fiber clusters 
was taking place, two cellular telephone franchises were being awarded to the state. One, Kansas Independent 
Networks Incorporated (KINI), was a consortium developed by some of the independent telephone networks. Most 
are located in central and western Kansas. Kansas Cellular decided that by connecting the individual cell sites 
with twelve-strand fiber, adequate bandwidth for growth would provide communication highways for education, 
business, and healthcare agencies. The construction of a point-of-presence on the campus provided the fiber optic 
tronk line needed to connect the school clusters to FHSU. 

In January of 1992, courses in special education and philosophy were offered to the Interactive Consortium 
Academic Network (ICAN). ICAN's fiber optic analog circuit terminated at the Rural Telephone Company "pop" 
in Victoria, 1 1 miles from the FHSU campus. At that time, ICAN was the newest fiber optic cluster in the stale 
and was of'the first to embrace the idea of using off-hours as a way to offer college-credit classes. During the first 
semester, enrollment was sparse. However, the university administration determined it would take time for ITV lo 
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be accepted. A firm commitment was made to maintain an aggressive posture in the development of new classes 
and to offer degree programs that students could take without having to step foot on the Hays campus. 

The following fall, KINI completed a fiber optic trunk line south to Dodge City. As one of the largest 
population centers in the service, Dodge had no upper-division college-level courses. It also had the distinction of 
being the first fiber optic cluster formed in the state. The High Southwest Plains Network (HSPN) included sites as 
far south as the Oklahoma border. It was immediately followed by the A-Plus network. There is a combined total 
of twenty sites in the south-central Kansas area. Prior to completion of the interactive gateway, FHSU approved 
the funding necessary to complete the construction of an interactive electronic classroom. At present, FHSU is an 
associate member of HSPN and is offering a diverse schedule of upper-division undergraduate and graduate 
courses. The enrollment of the ICAN and HSPN ITV courses steadily increases each year. 

Partnerships 

Anderson and Kimber (1991) wrote about meeting continuing education (CE) needs or rural nurses. They 
pointed out that rural nurses see themselves as cut off from the professional mainstream. The primary author 
determined to create an ITV environment that would make the nurses feel comfortable. Shomaker (1993) states it 
is vital to modify student and faculty support services on campus for use with distance learners. The challenge was 
to determine what to modify and what support services were needed. Authors such as Clark (1993) note the reward 
for meeting ITV challenges is satisfied distance learners. Such was the goal of the primary author as she planned 
multimedia lectures adapted for ITV. 

Considerable effort had been expended on off-campus partnerships to support ITV delivery; little thought 
had been devoted toward on-campus partnerships. The ITV environment at FHSU was planned to use multimedia 
such as Amiga graphics, videotape clips and special effects that were to be generated and/or supported by a 
technician. This plan, however, would have disrupted course development cycles for some faculty, including the 
primary author. Partnerships began to form. 

Faculty are used to lecturing in a traditional environment (that looks like a classroom). The technology 
orientation of the ITV support staff caused the electronic classroom to look and function much like a TV studio. 
The Amiga computer was placed in the control room in the back, for technician use. The faculty bunker had no 
computer and limited space to put faculty notes, props, etc. A request was approved by the Provost and sent to the 
Computing Center to install a computer for the faculty "bunker" (front of the classroom); space was made and a 
386 installed rather quickly. An ISN connection to the mainframe was then requested and installed. The 
Telecommunications staff purchased and mounted a notes holder to assist with lectures. The bunker was then 
much easier to use. 

Moving from traditional to ITV lectures requires a mapping process: a TV-like storyboard that blocks the 
time and gives the technician specific instructions. Faculty are not used to handing a plan to someone before every 
lecture. They are also not used to using multimedia in a technician-supported electronic classroom. Multimedia is 
the delivery of information, usually via personal computer, that combines different formats, including text, 
graphics, audio, still images, animation, and motion video (Miller, Sayers, Reeve & Kasten, 1991). If faculty get 
involved in creating their own multimedia presentations, the outcomes are strengthened. The creative process is 
further strengthened for some faculty by developing "on the fly" but support is still necessary and important. 

The electronic classroom was used to turn multimedia lectures into ITV presentations, with its SkyCam 
(ceiling camera), 386 computer and computer-based 16 mm. slide projector. Fred Hofstetter (IBM Scholar, 
University of Delaware) points out that faculty tend to get the lecture ready at 10:30 the night before. Ultimately 
the faculty in charge of the course is responsible for the lecture. If there is a last minute handout, one copy can be 
faxed to each site from the faculty bunker. The activities may sound easy, but they take numerous partners to make 
it all happen. For example, faxing is done by the classroom technician. To access the on-line library catalog 
during class, the Computing Center's ISN connection is available. It allows access to the E-Mail system 
(OfficeVision), Internet and other computer mainframe features. Faculty may demonstrate these resources on-line 
across the plains of Kansas for a different kind of multimedia lecture. An additional Computing Center resource 
that supported ITV was the computer-based voice mail system (interactive audio). Students who th-j"ght of 
questions after class left digitized voice messages that were answered in real time by faculty. 

One unanticipated area was requests from students for lecture videotapes. When a student missed class, for 
example, due to illness, she wanted to experience the ITV lecture (there were no men in the ITV classes). The 
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reserve videotape at the FHSU library did not help a distance learner. The logistics of faculty management to mail 
videotapes was time consuming; eventually, students were asked to arrange for videotaping to takf; place at the 
local high school site during class. All sites had a VCR but not all of these were in working condition. Local 
partnership work was needed to get all VCRs working; the results were satisfactory. 

Technology-Driven vs. Needs-Driven 

The integration of technology into traditional teaching schemes is one that must be accomplished with 
sufficient planning. The design should be scientifically developed and installed after appropriate assessments have 
been completed, marketing strategies devised, and needs evaluated. 

The importance of instructional design cannot be overstated. The reason that technological integration 
should grow out of an instructional design model is that assessment of the learning context, needs analysis and 
student evaluations (the heart of traditional instruction) are not less important with ITV. Instructional design 
rather than technology design is the standard by which all interactive systems should be measured during 
preplanning, construction and integration into the academic mainstream. 

Increasing the synthesis of information does little unless faculty understand that intelligence models do not 
change but require a new set of parameters in which to operate. Introducing new presentation forms could actually 
impede the process if learning strategies do not take into account the strengths and weaknesses of the media. The 
technology should become transparent to both the faculty and the student. Developing a technologically 
sophisticated delivery system provides little benefit if the message that is delivered is unimportant, distorted or lost 
in a world of technical jargon. 

The development of learning objectives is the area of planning that provides a foundation for all other units 
of the instructional design scheme and technology planning. A learning objective is an operational description of 
an anticipated change in the behavior of a student after exposure to instruction. It states exactly what it is that the 
faculty expects the student to be able do as a result of having learned something (Cyrs & . ? ith, 1990). 

Therefore, it is important that the learning environment design flows from: the learning context, student 
assessment, task analysis, learning objectives, and self-assessment. These instructional design components should 
be implemented before the cameras, computers, and television monitors arc installed. Faculty at FHSU attempted 
to do this, but got bogged down in the technology itself. A course evaluation tool created by the Instructional 
Media Committee was one of the few components prepared ahead of time. 

The situation as it occurred at FHSU supports the need for design components. Much attention was placed 
upon the development of fiber optic connectivity, the construction of the electronic classroom, and the attempt to 
connect to as many distant sites as possible (seven sites can be connected concurrently for one class). Little 
attention was devoted to faculty development, instructional design, or curriculum development. The system 
struggled for over a year before a scheme developed for faculty training and support. The President established an 
Information Management Task Force to study information problems (two of the authors, M. R. Hassett & M. 
Leikam, were task force members). Out of this study came the new Center for Teaching Excellence and Learning 
Technology. 

How One Faculty Adjusted 

A short videotape " Longitudinal Study: ITV-ship " (Hassett, Leikam & Jarmer, 1992) shows representative 
segments of electronic classroom instruction from two graduate nursing courses, covering a one-year period. The 
electronic classroom was intimidating at first; one must pay attention to the subject matter and the new technology. 
The primary av*hor evaluated her performance by keeping a weekly anecdotal record. At first, this was painful. 
However, problems were identified early-on for which partnerships could assist. Some mistakes shown on the 
videotape include: 

• had to use chalk board because did not know to bring colored pens and paper for SkyCam, 

• did not look at camera, 

• wore wrong color (chromakcy showed text across abdomen), and 

• did not control lecture notes pages. 
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Gradually, performance improved and there were fewer problems with delivery. Feedback from the review 
of videotapes and the student feedback was helpful. (It would have been good to also receive feedback from the 
assigned technician.) Some positive behaviors shown on the videotape include: 

• controlled lecture notes, 

• looked at camera more often, 

• relaxed and smiled more, 
wore right colors, 

had pockets in clothing (for wireless microphone), and 
called on students by name (instead of saying, e.g., "Dodge City"). 
It is very difficult to recall names from several sites and to recall which st udent is at which she, all the while 
keeping calm! Therefore, an old-fashioned seating chart was used with the names of students for each site (see 
Figure 1). The chart was used during each lecture to help keep the class active (rather than seeing a talking head 
faculty). 

A question that came to mind was: Should ITV episodes be called 'lectures"? 'Video forums" may be a 
more appropriate term.. 
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Figure 1. ITV Seating Chart, Developing Nursing Theories, Fall 1993 



Benefits for Faculty 



Tools used to evaluate were: a questionnaire written by the campus-wide Instnictional Media Committee, 
and anecdotal note pages ibr faculty (documenting what appeared to work and what did not and validating later). 
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This method was used for four semesters (the first semester course did not go off-campus but the ITV classroom 
was used as if it were). 

If faculty do not perceive benefit to themselves as busy people, technology may be ignored. Some faculty 
are skittish about offering an ITV course; one recently said "I don't want people across Western Kansas to see thai 
my new course isn't well developed.* 1 Faculty benefits include: 

• decreased "windshield time" driving to and from distant sites, 

• immediate feedback to m&from students, 

• satisfaction at being able to do a better job, and 

• preparation for the future. 

The primaiy author found the biggest benefit was the way ITV helped develop belter lectures than ever 
before. There was not much space for informal creating of lecture material. ITV forced her to be a better 
teacher/facilitator. (A regular classroom now seems dull.) Collcf iality among the partners was an additional 
benefit. 

Benefits for Learners 

Student uses parallel those of faculty and benefits include: 

• decreased "windshield time" driving to and from the university classroom, 

• immediate feedback to and from faculty, 

• immediate feedback to and from other students, and 

• preparation for the future. 

The immediate feedback on video ;s startling in the way it facilitates student learning. A picture is worth 
a thousand words. Feedback from other students was important; one international student particularly seemed to 
benefit (her native language is Chinese). Her participation level was higher with ITV than was usual for a regular 
class. Student networking was facilitated, particularly among those at the same site. 



Transformation Stages 

The electronic classroom is the last of the transformation stages (Hassett, 1993). Opaque was our first stage 
of transformation in teaching with technology: it means impenetrable by light, or dull. Technology in the past was 
opaque; one could not see through it because of barriers . Opaque examples include: blackboards, low intensity 
overhead projectors with light bulbs dying on a regular basis, and very little use of color. 

Translucent means transfusing light but diffusing it to cause images to become blurred. Today, we are 
primarily in this stage, in the distractor mode. We partners tried laying a portable computer under the SkyCam.in 
an early ITV class to show output (video feed scrolled and crawled). One piece of technology changed that: when 
the genlock was acquired, the scrolling stopped and output to the classroom monitors became clear as a fine TV 
set. . . that is, it became transparent. 

Transparent means capable of transmitting light so that objects on the other side can be seen. What is 
wanted for the learner is to see through the technology without even noticing the technology is there, in the enablcr 
mode. A graduate student who uses the Dodge City site shared after the second week of ITV that she did not 
notice the video camera ainy more: *i felt like 1 was in the classroom with you. M 



Planning for Future ITV Courses 

Only a few faculty are presently doing ITV. How should the multimedia lectures of today be developed so they 
will also work for future ITV? The primaiy author learned to stoiyboaid each lecture for the technician in the 
control room. This was new and required additional time but strengthened lectures. How many faculty have to 
show a detailed plan to anyone before doing a lecture? For ITV, one docs it every time. Faculty do not have to 
wait for ITV to stoiyboard; the technique would work for any course. 

It took the primary author three to six hours per electronic classroom scheduled clock, hour to prepare an ITV 
class. FHSU compresses two ITV clock hours each week for a three credit hour class; therefore, it took 9 to 12 
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clock hours per week to prepare (for an already developed course). Initial outlay of time is extraordinary! Tips for 
ITV include: 

• have back-up material ready (e.g., keep extra material nearby in case something goes haywire), 

• minimize distracting mannerisms, 

• send handouts 10 days ahead (use a lecture packet students purchase from the campus book store if 
possible), and 

• do not be too serious (smile more)! 

And if the technology fails or acts up, do not apologize-IMPROVISE. ITV prepares one for the future when 
this type of communication will be common. Examples could include: 

• ITV consultation for a Family Nurse Practitioner in Ransom, Kansas and a physician at The University of 
Kansas Medical Center, 

• ITV nurse administrator meetings for consortia activity, 

• ITV continuing education for rural clinical nurse specialists, 

• ITV statewide professional society meetings, and 

• ITV consultations with a master teacher at FHSU for a community college faculty. 

A number of the graduate students taking ITV classes had a computer available to them. In the future, rural 
students may be required to have their own computers. Many urban universities already require this. 

We found that success for ITV in the sparsely populated FHSU service area requires offering complete 
degree programs. Other examples include classes for: teacher rccertification, nursing relicensure, and workforce 
retraining. The Master of Science in Nursing program has been highly successful because of demand and a 
distance learning delivery that meets the needs of place-bound working nurses. FHSU was not successful in 
offering courses that did not fit into a planned curriculum or that could not result in the completion of an 
undergraduate or graduate degree. 

Support for delivery of ITV courses is complicated. Merging the existing technologies of multimedia with 
ITV required partnerships. The departments used to support or augment merging included: Nursing, Computer 
Information Systems & Quantitative Methods, Information Networking & Telecommunications, the Computing 
Center, and Forsyth Library. These partnerships strengthened the ITV offerings and the relationships among the 
partners. 
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Abstract: This paper reports the results of a joint venture by the Society of 
Actuaries and Penn State University to investigate the effectiveness of a computer- 
based multimedia platform for the continuing education of actuaries. The goal of 
the project was to produce a prototype multimedia module for continuing 
education and to have it evaluated by practicing actuaries. 

The participants* reviews of the program were very favorable. The majority 
indicated that the multimedia presentation was worthwhile and that they would 
recommend the multimedia delivery system to their employers. 

Introduction 

Last fall, the Society of Actuaries and Penn State University undertook a study to investigate the 
effectiveness of a computer-based multimedia platform for continuing education. Among other things, 
multimedia was being examined as a potential means to overcome the limitations of instructional video tapes. 
The goal was to produce a prototype multimedia module for continuing education and to have it evaluated 
by practicing actuaries. 

A current taxation issue was chosen as the topic for the prototype and the Chief of the Actuarial Branch 
of the IRS was chosen as the content expert. The prototype was then tested at two Society meetings: the 
annual meeting in Washington, in October, 1992, and the spring meeting in San Diego, in April, 1993. The 
advantage of using Society meetings as test sites was that testing and evaluation could be done by potential 
users without their having to purchase equipment or incur any cost. 

The participants' reviews of the program were very favorable. The majority indicated thai the 
multimedia presentation was worthwhile and that they would recommend the multimedia delivery system 
to their employers. 

This article discusses some advantages of a multimedia platform over instructional video tapes and 
reports on the opinions of the actuaries who participated in the multimedia sessions. 
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Why is Multimedia a Good Educational Tool? 

Everyone studies and learns in a different way and at a different pace. Multimedia is a good educational 
tool because it is well suited to accommodate these characteristics; it allows users to proceed through the 
material in the way that best suits their needs (Jonassen, 1991). For example, users do not have to proceed 
linearly through lessons. They can skip previously learned lesson segments, move forward at their own pace, 
and review when necessary. 

Comparison with Video Tapes 

Video tape, while appropriate for many instructional applications and situations, is limited because it 
is not interactive and it presents information in a continuous, linear fashion (Hannafm & Peck, 1989). One 
purpose of this project was to examine multimedia as a potential means with which to circumvent these 
limitations of instructional video tapes. 

Since video tape is not interactive, there is no opportunity for users to provide input. Consequently, 
performance cannot be monitored and injunction cannot be adapted to the user. For example, there is no 
way to determine whether a user misunderstands a concept and needs review or understands a subject and 
should advance to the next lesson. 

Video tape does not allow instant access to information. Users, when deciding to skip a previously 
learned topic cannot do so without fast-forwarding or rewinding the tape, which is a relatively slow and 
inefficient process. The instant access to information available in a multimedia system should encourage 
learners to go back and review a trouble spot, whereas the difficulty of finding the right spot on a video tape 
might discourage them from doing so. A related problem is that in the time it takes to search for a specific 
lesson segment, the reason for the search may no longer be clear. 

Advantages of Multimedia 

The multimedia prototype provides several important advantages to the users. 

Multimedia gives users the ability to jump to a specific topic at will. Users are not subject to a 
continuous flow of information from beginning to end, but rather they have instant (random) access to and 
control over the information (Magarry, 1988; Jonassen, 1991). The result is that individual learning rates 
and styles are better accommodated. 

Multimedia facilitates interaction with the lesson. In addition to choosing how to proceed through the 
lesson, users can respond to questions, manipulate objects, and input numeric values into tables. These 
activities are included in the lesson to proiaote learning. As many researchers have noted, the acquisition 
and retention of information is more likely to occur when the learaer is actively involved with instruction 
rather than being a passive recipient of it (Hannafm & Peck, 1989; Cennamo, Savenye & Smith, 1991). 

Testing is design to ensure that important concepts are learned by all program users and to alleviate the 
fear of failure often associated with conventional instruction and assessment. For example, review questions 
are presented after each content segment. As users respond to questions, the multimedia system monitors 
their responses and coaches them to formulate correct answers. Incorrect answers are detected by the 
system pjid feedback and remediation given, to help the user derive a better understanding of the concept. 

Finally, multimedia lesson materials are integrated into a single computer-based platform. This gives 
users control over a host of instructional resources and make all the advantages of multimedia discussed 
above possible. 
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Evaluation of Ihe Multimedia Platform 

The prototype was evaluated by 21 actuaries at the annual meeting in Washington and by 26 actuaries 
at the spring meeting in San Diego. These numbers so not reflect the people who signed up for the session 
but could not be accommodated. In San diego, for example, 55 individuals were wait listed. The 
assessments at the Washington meeting (see Shapiro, Gibbs, & Hall, 1992), were consistent with the 
favorable assessments given in San Diego and reported below. 

At the San Diego meeting, participants were asked to assess the effectiveness of the multimedia 
presentation format by responding to several statements. Participants rated the program on a 5 point scale, 
where 5 corresponded to Strongly Agree and 1 corresponded to Strongly Disagree. A summary of the 
responses to these questions is shown in Table 1. 

Table 1 



Participant Reactions to Presentational Format 



Reaction Statement 


strongly 
agree 




Neutral 




strongly 
disagree 




5 


4 


3 


2 


I 


The presentation was well organized. 


71% 


29% 








Visual materials were appropriate and 
enhanced the presentation. 


83% 


17% 








This was the appropriate format for 
presentation of this topic. 


58% 


25% 


17% 






My time was well spent attending this 
session 


38% 


46% 


13% 


4% 





Numbers may not add to 100% due to rounding. 



In addition, participants were asked to answer the questions shown in Table 2. For comparison, the 
Washington results ( N D.C. H columns) are shown in addition to the San Diego results ( N S.D. N columns). The 
results show that the majority of participants at both meetings responded favorably to the delivery system. 
Over 90 percent en;oyed the delivery system and thought that other programs should be developed, and 80 
percent said they would recommend the delivery system to their employers. 

Table 2 



Participants Overall Reactions 





Yes 


No 


Sort of 


Indifferent 


No Response 


D.C 


S.D. 


D.C 


S.D. 


D.C. 


S.D. 


D.C. 


S.D. 


D.C. 


S.D. 


Did you enjoy this delivery system? 


90% 


96% 




4% 


5% 




5% 








Should other programs be 
developed using ihis delivery 
system? 


90% 


96% 










5% 


4% 


5% 




Would you recommend this 
delivery system to your employer 
for use within the company? 


81% 


78% 


5% 


13% 










14% 


9% 



Participants had an opportunity, after each session, to write their reactions to the prototype. A sumnary 
of comments from both meetings is listed below: 
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"It involves the student (me); it is participatory and this causes it to be more enjoyable and more 
effective [then] lectures." 

"I like it because I could control the time/instruction. If I missed something I can rerun it again. Its 
Great!" 

"Quiz is helpful in sustaining attention. I like the fact that you could replay something you didn't 
understand the first time." 

"I found the multimedia, individually oriented, quite interesting and well done. I can go at my own 
speed." 

"Multimedia engages multiple senses." 
"Fantastic Graphics!" 

Conclusion 

The attractive characteristics of multimedia - such as, self-directed and self-paced learning, lesson 
interaction, dynamic testing and instant access to information - combine to create an ideal learning 
environment and help to overcome the limitations of instructional video tapes. From the experience at both 
the Washington and Svx Diego meetings, it appears that this environment is well suited to the needs and 
preferences of the modern actuary. Therefore, it seems inevitable that multimedia will play a significant role 
in the continuing education of actuaries. 
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Abstract: This project is in progress and concerns the computer supported teaching 
and learning of " Finance " in courses at university level ( Faculty of Economics and 
Business Administration School ).The design development strategy of the project is 
based on a joint venture by a variety of experts and students at the level of degree 
dissertation . One of the outputs of the project is a running computer-based tool for 
P.C.'s (hypertext software-based tool). This tool has two "modes" of use: The 
"teaching mode" , i.e. the mode lor the speakers lectures (Professor of "Finiinee"); 
and the "learning mode" , i.e. the mode tor the Learner, the free use on the part of 
the student. The theoretical foundation of the model, its design and its first results 
are discussed. 



Introduction* 

As explained in the abstract of this paper we will de;d with a project in progress. Within this project a given 
prototype running on Macintosh computers has been realised, and a certain number of users has already used 
the program itself. We are now hoping to realise a more widespread use of it for many students during die 
present academic year . According our experience and predictions, we expect to collect new and interesting 
results in order to enhance this product, to improve the model and above all to enrich our scientific knowledge. 

The word "system" will often be employed in a broader sense to indicate an arrangement of individuals mid 
things so related or connected as to form a unity or orgimic whole. The term "project" will be used to indicate 
the planning state or condition of the system under discussion. This arrangement of tilings will obviously 
include hard and software , i.e., computers and above all software prototypes as output of the human activities 
put into effect during the planning activities. The word "model" will be used in its precise meaning of 
representation and design (Bussolin,l<J92). 

The name of this project (Hyper Finance), as may easily be understood, comes from die combination of the 
two words "hypercard" and "finance". In fact, this prototype has been revised in the hypercard environment, 
has been written with hypertalk language, and its purpose is the teaching of Finance within the courses of the 
Faculty of Economics of the University of Torino, and the courses of the Business Administration School of that 
University . 

Part 1, Theoretical and conceptual foundation of the project. Some 
considerations on its guidelines and its principles. 

Before dealing with this project in a more analytical way, it seems to be very important to reflect on its 
foundations and to make some considerations which lie at the base of the whole work and which justify, we 
may say, the choices made throughout the process of its construction. These reflections and considerations will 
be set out in random order, therefore without precise priority , as Uiey belong to different orders of thought and 
to different scientific fields. 

'Hie first point from which many considerations follow, might be titled: the "human side of die sciences of 
design and of the sciences of projects". The idea of utilising the techniques of hypertexts and a given software in 
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order to realise a prototype running on a personal computer tor the teaching in a business administration course 
is not really a new one: some realisations exist in this field, even if they are not numerous (Ruggero, 1993). 

It seems to us that, given the above mentioned considerations, it is worth remembering how the "human 
side" has played and will play an even more definite role, a role that strongly influences the success of a project 
of such a nature. As a key to the interpretation of this side, the whole thinking of H. A. Simon is significant, 
particularly certain of his works which will be cited from time to time. 

There is often talk about the barriers to the penetration and implementation of the new instructional 
technologies (Postero,1992), as there is, in general, about the use of this computer technology in the world of 
human organisations, the world of profit and non- profit organisations. 

During the course of this project, we have frequendy noticed how in the field of computer-based education, 
as one moves towards higher levels of teaching, one encounters specific and perhaps greater difficulties, and 
specific and perhaps greater barriers of a different nature. We have noticed how the applications in the different 
fields of science are not uniformly distributed (Ruggero, 1993). Particularly in the field of Economics, of 
Administration Science, the implemented tools for computer- based training and education are extremely scarce 
( i.e. in number ) for the University levels in comparison the remaining fields. An investigation concerning the 
why's of this state of affairs probably might lead us out of the precise domain of exposition that we have 
undertaken. Nevertheless, some considerations may be offered. 

Within the domain of the literature concerning the use of information technology for education it seems to us 
that the two aspects of a single problem are not taken into precise account , mainly as regards the computer 
tools as a teaching aid: the Teacher's aspect and the Student's aspect. Most attention is paid, mid most scientific 
considerations are dedicated to the sphere of behavior and the knowledge domain of the Learner, i.e. the person 
who benefits, the person who makes use, as receiver, of the scientific communication based on mid supported by 
the information technology. Normally the "hows" of this learning process are investigated in respect of and 
according to the changing of many conditions, in conformity with the technology contingently being used, in 
accordance with the modes of its use. One of the most important purposes of these researches is that of 
improving the "quality" of learning (Bussolin, 1992). According to our thinking, it seems that the theme for the 
development of this technology for teaching, for education, concerns equally, in the first instance, not only ttie 
learning processes of the Student, but also the cognitive processes of the Teacher who must, or wants to or may 
use this technology. 

As a first approximation, it seems that the cognitive processes involved or to be developed for the Teacher 
are by far more complex and offer greater barriers and difficulties than those encountered for learning on the 
part of the Student. Normally, for the Learner it is the "quality" of learning which promotes the use of this 
technology (without for the moment analysing the complex meaning of the issue "quality of learning"). For 
the Teacher, the University Professor , a dual purpose exists : the quality of knowledge that this person has to 
transmit is involved. In fact , the use of this technology carries with it , often, the task of reshaping the 
knowledge to be transmitted, the task of systematising the universe of the Teacher's own knowledge, the task of 
redefining the modes of communication and transmission of his/her own knowledge, etc. etc. 

As already mentioned, there are many likenesses among the processes of use of this information technology 
within organisations in general ( see projects of office automation, of decision support systems, of large 
database systems, etc. ) and its use for educational purposes. One of these likenesses or analogies lies certainly in 
the {act that the side effects of its use always lead us to a redefinition of the state of affairs of the system which 
is object of the computer-oriented application. 

As the educational level increases, always remaining in the field of Economic and Administration Sciences, 
particularly at the university level, the representations of the knowledge domain to be transmitted become 
more and more subjective representations, therefore more linked to the personal interpretations that the Lecturer 
makes for his own discipline to be transmitted. These representations are more and more connected to his 
personal education culture and background, to the shapes and to the processes according to which this 
knowledge has been generated in time, to the dynamics of the new elements that the Lecturer decides to add to 
his own system of knowledge and to the dynamics of the elements that, are to be abandoned. These cognitive 
studies concerning the processes of generating and storing the knowledge of the Lecturer, must also take into 
account the propensity of the Lecturer to use this very technology. 

Furthermore, the forms of transmission of this knowledge are extremely varied and influenced, to a certain 
degree, by the organisational environment within which this educational process is being csirried on. These 
shapes of transmissions are closely interwoven with the factual conditions of this environment, with the 
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contingent state of the information technology at hand and the contingent state of the very organization within 
which a member of it, i.e. the Teacher, acts and behaves. On this subject, we might better say that the project 
under discussion in this paper, and generally speaking projects of this kind, must keep separate the two 
following aspects: the cognitive aspect of the Teacher who wishes to use or to improve the use of this 
information technology, and the aspect of the organizational influences that are the premises to the initial 
decision to use it, and the premises to the subsequent decisions which will be made when the instructional 
system is brought into being within the educational organization. 

The employment of this technology always involves not only the Teacher or Lecturer who uses it, but the 
whole university organization or, generally speaking, the whole educational organization, within which the 
process is being developed. With traditional teaching, the cognitive premises of the teacher and the 
organizational influences upon him are very much more indefinite, and probably they have been less thoroughly 
investigated and have given rise to fewer scientific observations in comparison with what happens now , since 
the educational technologies have come onstage. The study of Computer Based Learning may give fruitful 
results, both scientific and practical, if we consider the Learner and Teacher as members of a specific 
organization . In this sense the process of transmiss'on of knowledge may be investigated as a cognitive 
process belonging to the organizational premises for decision making ( Simon,1947; Kleindorfer, Kunreuther 
& Schoemaker s 1993). 

Finally, let us stress the importance of the project aspect, i.e. the side of the experimental phase for the 
construction the Computer Based Learning prototypes, unlike what happens for traditional learning. We have 
for a CBL project , in fact, a variety of experts who participate in the project: not only the expert on the 
knowledge domain, and the expert or experts on the information technology, but also the users (Learners and 
Teachers) who benefit from Uiis project and who may give useful answers and stimuli during the process of 
experimental use of the software prototypes being constructed. Furthermore, during this project phase, we have 
the possibility of planning the roies that will be played by the human "agents" and the information technology 
"elements" according to a variety of modes and shapes, etc. etc. ( see Postero, 1993 and works referenced ). 

Part 2. The development of the project. Coasideratioas around its foundation 
elements and its first results. 

Let us consider now, more closely, the realisation of the project, the elements which compose it, its first 
results. This second part, also, will be set out according to a list of points in random order which are not in a 
defini te order of priority. 

As previously said, the purpose of this project is to develop the teaching at a university level through a new 
technology for a course in a Faculty of Economics. The principal target is to support the motivation for the use 
on the part of the Lecturer ( a University Professor ) and to improve the quality of learning on the pjtrt of the 
students. The construction of these models, and the software prototypes deriving from them is inspired , from 
the scientific point of view, by the principles of the Artificial Sciences (Simon, 1969) and specifically by die 
principle according to which die whole project and the model taking shape within i: , itre so to speiik "molded" 
by the purpose lying at the base of the whole construction. 

The followip^ observations may fall within the compass of interfaces as a meeting point between the 
"inner 44 environment, i.e. die knowledge of the Teacher to be transmitted, and mi "outer" environment, i.e. the 
learning process of the Students. (Simon, 1969). What has been called 44 model" in our project, might better be 
understood as the design of such an interface. In the course of this second part of the paper we will be more 
specifically concerned with the design of such an interface. 

Given the purpose previously and specifically mentioned, a hienirchicid shape has been chosen for the 
hypcrtextual structure of concepts to be transmitted , discarding the hypodiesis of other solutions well known in 
the literature ( Conklin, 1987)(Van Dike Parunak.1989). 

The hierarchical shape is the most appropriate to realise a gradual process of knowledge so that the Teacher 
, during the first steps of the project, may obtain a greater confidence toward the use of the computer-based tool 
. The strongest cognitive representation that a Teacher may have, as regards the subject or discipline taught, 
and when this teacher has not yet utilised the hypertextual technology, is a hienirchicid representation of die 
concepts belonging to the knowledge domains of his subject or discipline. This issue is , of course, based on 
our personal and empirical observations from the project carried out and it would be very interesting to evaluate 
this assumption through more scientific experiments . 
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^Tnfra reason similar to the one previously shown, we have used the "metaphor" of the book . The Teacher 
and the Student find, in the first images on the screen , the design of a series of books named Riskamtlisys, 
Exercises, User manual, md starting from each book, the navigation may begin . At the end of each book a 
series of tests may help the Student to measure the degree of learning he/she has attained ( Giordmio.lOW). 

The cognitive problem of initiating the Teacher in the use of this technology must be solved for the 
development of the instructional technology itself . The traditional relationship existing between the Teacher 
and the book, this last being intended as a hard copy containing his own concepts, and the relationship among 
the users and the sequential flow of concepts, are very strong traditional cognitive relationships . Only after 
having learned how to use a hypertextual computer-based tool, will the user accept a new concepts 
representation based on a free navigation. Under these circumstances, it will therefore be possible to 
"rewrite*' the whole content of tne user's knowledge domain through the concepts composing it, organising this 
domain by a struciure based on new shapes of links and new modes of navigation . 

The same considerations made with regard to the Teacher's side, may be made now, "mutatis mutandis", 
with regard to the graduating Students' side, i.e. the members who have participated in the project and in the 
construction of the model itself. These members may be considered as potential users: they have undertaken a 
joint venture as bearers of computer science expertise and bearers of administration science expertise . They 
have shared the purpose of the project and they have given rise to a "mix" of expertises ( Spool, 1993). 

Hierarchical structures of concept, which may be thought of in a logical sense as trees, have been 
constructed . The elementary concepts ( leaves of the trees) represent the minimal units of the model. These 
minimal units are in a given correspondence with a succession of images on the screen. This analytical 
approach to representing human knowledge might be considered , " a posteriori", a reductionist approach, 
while the holistic approach might be considered as the one needed for the generation of links adopted within the 
structure of the model or during the computer-based user interaction. 

As previously said, the metaphor of the book has been used in order to individualize the root of each tree 
composing the "forest". Each tree individualizes a book, the series of trees individualize a forest. The "forest" 
may be imagined as the whole course that has to be transmitted to the students. 

The first idea of the navigation that may be understood consistently at the beginning of the project, is the 
idea of a sequential navigation; that is, to read forward and backward the leaves of our trees juid to sesirch for the 
arguments through a hierarchical navigation in order to find the various books, chapters, piuagraphs, etc. 

Upon the base structure so built it has been possible to initiate (I repeat,"initiate") the "holistic" approach 
previously mentioned, that is, an approach aiming at the construction of a system of non-hienirchically 
structured links . This construction is strictly connected with the inlterface support , i.e. the support giving 
evidence to the user on the computer screen of the commands ( icons, pull down menus, symbols, etc.) which 
allow the user a different navigation from the sequential one seen previously. 

Before speaking about this navigation we would prefer to make brief reference to the problems which arise 
for the construction of the user interface. Of particular interest for the project have been the "hows", the modes 
in which the user interfaces have been built, therefore the construction of each screen form. This concept of 
interface, previously mentioned, is here to be understood in a wide sense, therefore also with regard to the use of 
a given language, not simply natural, but also symbolic and, above all, a language based on icons, buttons, pull 
down menus etc.. 

This last kind of language has a great cognitive potential and , in our project, it allowed us to create specific 
communication means on the part of the Teacher, and it permitted us to offer specific leiirning means to the 
Student. We have previously mentioned the "hows" in which the user interfaces have been built in order to 
suggest that the use of this "non natural" language has been one of the first steps that the project team (students 
in Economics and students in Computer Science) have learnt in order to construct these very user interfaces. 

It seems to us very important to remember that in a project team, of the nature under discussion, the 
interface language must reach a consistent and precise level of definition , it has to be learnt and shared by its 
members and therefore it must be used by them according to a definite system of stiindiirds. This last 
mentioned aspect or moment of the scientific investigation is certainly crucial for the whole success of the 
project, if we think that the builders themselves are the potential users ! 'llie concept of "bounded rationality" 
(Simon, 1945) has been inextricably interwoven with the process of design realised by the members of the 
project under discussion. The "rationale" involved within this process must be better investigated i ,% order to 
give a foundation to the construction of such interfaces. 
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The first important function that we faced wjs that for a traditional reading, iherefore the use of buttons for 
the sequential navigation forward and backward, the map for the cognitive orientation and the icon for the 
history in order to lessen the cognitive disorientation. 

Furthermore we have implemented: 1) buttons for the "static" and the "dynamic" deepening of concepts in 
order to descend below the level of the traditional knowledge contained at the level of the "leaves" (pages 
containing the concepts); 2)the hot words and a glossary in order to find the meaning of the most important 
terms; 3) the pull down menus to create personalised paths of teaching , on the part of the Teacher (the Teacher 
mode), and to create personalised paths of learning, on the part of the Student (the Student mode) 
(Normanl991). In fact , according to these last pull down menus, the Teacher may choose , after a previous 
definition, a given path to teach his concepts within a given lesson, and each Student may define one or more 
learning paths for a re-visitation of his own previous learning processes within a given temporal stage of his own 
learning . 

The prototypes constructed thus far allow a free definition of the learning paths on the part of the students. 
This opportunity involves the problem of the learning control embedded in the model : when and if to leave the 
control of the learning flow of events to the Student or to the model in use. At present, the student has a free 
opportunity to go into the depth of the concepts represented in the screen forms (leaves mid nodes), he can make 
free choices to navigate through the screen forms etc. . 

We must not forget that we are in the initial stages of the project. Only in a subsequent phase of it , when 
more empirical results coming from a more exhaustive application are at our disposition , will it be possible to 
"superimpose" on the model some structures or modules which might allow a >;iven kind of control, a given kind 
of diagnosis about the level of the acquired knowledge, etc. It is this last approach thai is followed by the ITS's 
(Intelligent Tutoring Systems) in which we find the tutorial module, the diagnosis module, etc. It seems to us 
possible to implement such a theory into the project under discussion (Poison, |1988). 

Many principles of the HCI (Human Computer Interaction) have been discussed by the project team in order 
to define the guidelines and the standards to be respected during the process of the project: unambiguous 
commanJs, their simplicity, appropriate choice of the objects that are always ;to be on the screen and those that 
may be rc-c?lled through the pull down menus. We have also taken into account the ergonoinic problem jind the 
cost-benefit analysis existing for the construction of an artifact of such a natuie: the degree of analysis iuid length 
(in words) of each concept, total number of screen forms per book, etc. etc. 

Conclusions 

Finally, we would like to conclude with some considerations which justify, "so to speak", this kind of 
approach where it is the interest in the human being rather than the artificial element, which predominates or 
prevails. 

It is interesting to devote some words to the "how's" the team has carried out the production of the prototype 
software. The prototype, we must not forget, is also a software product and the following considerations, even if 
briefly expressed, belong to the software engineering field. The traditional model, based on the traditional 
functional decomposition software methodologies, frequently used for the process of construction of softwjire 
on a large scale, has been discarded. According to this last approach, each phase constitutes, strictly speaking, 
an input for the subsequent phase, and above all, the final user only interacts with the construction process in the 
last phases, playing the role of a true user. In contrast, according to our philosophy, which we might call "a 
more cognitive philosophy or a more human-being-centered philosophy", the user, i.e. the Teacher and the 
Student, is a centra! figure that has to interact, from the very beginning, with the prototypes that the project team 
is going to realise during the course of the production process. This approach is obviously itllowed by the fact 
that the programming language has been object-oriented (Kim J. &Lerch F.J.,1992). We might truly assert 
that the engineering approach we have followed constitutes a mix of the "prototype approach" and the 
"incremental approach". In fact, as regards the first approach, the interaction with the user (the Teacher and the 
Student), has been carried on since the very beginning of the project phases, in that Teacher and Student are the 
central points of the project. According to the second approach, some special functions have been embedded 
in the prototypes since the first steps of the project and have been kept constant for (he whole duration of the 
project itself. 

When we deal with artifacts , we ought to keep in mind the "pcrsomil perspective" , that is the perspective of 
the user (both Learner and Teacher), and the "systemic perspective", that is Die perspective of the total system. 
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This last perspective puts together the artifact (the prototype), the user (Learner and Teacher) and the task or 
function (to learn). Certainly the Student and the Teacher who utilise the artifact will have a personal 
perspective where the cognition process may be enlarged or enhanced. The artifact in our case is specifically 
built for better learning . We agree, on the other hand, that the artifact may change the task and, principally, it 
may change *e mode of teaching on the part of the Teacher and the mode of learning on the part of the Student. 

According to a systemic perspective we may say that the artifact has changed or is going to change the 
functions of the user: for the Teacher new elements to be taught and new modes of teaching, and for the Student, 
above all, new questions that can be asked and new elements that may be learned. Finally* and this is 
certainly a straightforward "expectation", the artifact will create a new mode for distributing teaching and 
learning tasks within an educational organisation. 

As a "last but not least" consideration, we would like to stress a very interesting aspect or feature of this 
research, i.e. its project nature where the boundaries of rationality (Simon, 1945) involved in the project team 
have played a central role for the development of the model itself. These boundaries, according to our thought, 
are a central theme of investigation for the success of this kind of projects. 

The project team may be considered as a micro organisation or an organisational unit which carries out a 
particular task, and acts in order to reach a specific purpose or end (see the first part of this paper). Each 
member of this team has put into operation, within his/her job, a personal expertise about die knowledge domain 
of Finance to be taught and a personal expertise about the knowledge domain on the HyperCard environment. It 
has not been necessary, thanks to this human expertise, to have a tool for the Hypertext design, raiher the key 
problem has been to spread the knowledge of standards of the HyperCard language (Hyper l:»lk), to settle 
technical rules for the symbolic huiguage of icons,butlons, etc. within the members of the icsun (see above with 
reference to the language used by the team). In contrast with this approach, we often see some strongly 
structured approaches to the hypertext development based on the authoring on a huge scjiIc. 

Our impression therefore is that in this problem of CBL (Computer Based Learning) we have strong elements 
of influence of the contingent real environment We have > that is, contingent realities where what is prevalent, as 
in our case, are the organisational needs, the needs to improve the communication Junong the members of the 
group, therefore to build standards for a non natural language that will be used for die i4 iirt" of design . A 
creative component of each member exists, which mixes with the pure rationality of the prognunming 
environment. This whole problem constitutes a "terra incognita" for an interesting scientific investigation. 
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ABSTRACT 

The purpose of this study was to 
find out the extent to which 
computer-based interactive 
multimedia training (CBIMT) was 
developed or used by a select group 
of employers in Edmonton, 
Alberta, Canada. 

In February 1992, members of the 
Alberta Society for Human 
Resource and Organizational 
Development (ASHROD) were 
surveyed about their employers* 
development and use of CBIMT. 
ASHROD is an association of 
training, and organizational 
development professionals. 

Findings included the following: 
Twelve of the 44 respondent 
organizations developed CBIMT 
and 20 used it. 

Large organizations developed 
CBIMT to use and small 
organizations developed it to sell. 
Use of CBIMT varied as functions 
of both business activity and size or 
organization. 

Findings are discussed in 
comparison to findings of other 
North American studies. 



DEFINITIONS 

C omputer-based Interactive Multimedia 
Training (CBIMT) 

CBIMT is the use of CBT and/or CBIVT 
training environments to encourage 
learning through active processing of 
information (as opposed to the passive 
reception of information). A variety of 
other media can be used, including text 
and graphics. As the term CBIMT is 
relatively new, it was not used in the 
questionnaire. The component terms, 
CBT and CBIVT were used instead as they 
were judged to have more widespread 
meaning among survey respondents. 

Computer-based Training (CBT) 
CBT is the use of computers to deliver 
training. 

Computer-based Interactive Video Training 
(CBIVn 

CBIVT is the addition of video to 
Computer-based Training, 
training system. 
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U se of CBIMT in Can^ a 

Reports in periodicals indicate widespread 
use of CBIMT by large Canadian 
corporations (e.g. Duchastel, 1989). 

Two Canadian surveys included questions 
about CBIMT: a March 1989 survey of the 
use of computer-based technologies within 
Canadian service industries, 
(Communications Canada & Indtr y, 
Science and Technology Canada, .990) 
and a 1990 P ;iference Board of Canada 
survey of training in Canada (Larson & 
Blue, 1990). 

The Technologies in Service study 
showed that 17% of employers with fewer 
than 200 employees used Computer 
Assisted Education (CAE) while 28% of 
those with more than 200 employees used 
it. Use also varied by industry group (see 
Table 3). Overall, 22% of Canadian 
service industry employers used CAE 
(Communications Canada & Industry, 
Science and Technology Canada, 1990). 

The Conference Board Survey showed use 
of various training technologies to be 
between 6 - 15% of employers with 
substantial increases in use expected 
(Larson & Blue, 1990). 

Use of CBIMT in the United States 

Several U.S. surveys have included 
questions about use of CBIMT. Perhaps 
the mos* useful are those which have been 
conducted by Training magazine. 
Training , a journal oriented to the training 
profession, has conducted a survey of 
employer sponsored training annually since 
1983. Questions about CBIMT have been 
included in these surveys (Geber, 1989, 
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1991; Gordon, 1985, 1986, 1988; Lee, 
1987). Use of CBIMT among Training 
magazine survey respondents has grown 
throughout the period, with the level of 
use highest among large employers but 
growth in use highest among smaller 
organizations (Graph 1). Use of 
interactive video, however, has shown 
little growth, stalled at the 30 - 35% level 
among large organizations and at about 
15% among all respondent organizations. 

Hirschbuhl surveyed large U.S. 
corporations about their use of CBT. He 
reported that between 1985 and 1987 there 
was considerable growth in the use of 
CBIMT. He also identified variations in 
level of use among industry groups 
(Table 3). 

DEVELOPMENT OF CBIMT 

Information about the development of 
CBIMT is scarce and difficult to compare. 
Of 11 large Canadian employers featured 
in periodical articles between 1984 and 
1991 as users of CBIMT, 6 developed 
courseware in-house. In the 1990 
Conference Board of Canada survey it was 
found that 8% of respondents used 
authoring systems (Larson & Blue, 1990). 
Hirschbuhl found that in 1987, 6% of 
courseware was developed in-house, 24% 
custom developed and 70% was vendor- 
supplied generic software. 



THE STUDY 

The purpose of the study was to determine 
the status of CBIMT among a selected 
group of Edmonton employers. 
Organizations represented among the 
membership of the Edmonton chapter of 
the Alberta Society for Human Resource 
and Organizational Development 
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northern Alberta organizations. 

Omni Press. „ . _ , 

(ASHROD) were surveyed. ASHROD is 
an association of training, and 
organizational development professionals. 



common authoring systems used by 
developers. 



Fifty six ASHROD members were sent 
questionnaires. Forty four responded, for 
a response rate of 7^%. Respondent 
organizations included private, public and 
not-for-profit sector employers, and small 
organizations which developed and/or 
delivered training. 

The Questionnaire 

As well as demographic questions, the 
survey included questions asking: 

1. The extent to which survey 
respondents were developing and/or 
using CBIMT. 

2. Whether respondents planned to 
increase, maintain or decrease their 
development and/or use of CBIMT 
during the coming year. 

3. What authoring software 
respondents were using. 

4. What computer hardware survey 
respondents used. 

5. The job categories and types of 
skills is CBIMT was used for. 

6. The advantages and disadvantages 
of using CBIMT. 

FINDINGS 

Development of CBIMT 

Twelve of the 44 organizations developed 
CBIMT (Table 1). There were two kinds 
of developers - large organizations that 
developed for internal use and small 
organizations that developed courses for 
sale to other organizations. One large 
organization developed CBIMT for both 
internal use and sale (Graph 2 & table 2). 

f 

Quest and Authorware were the most 



Use of CBIMT 

Of the 44 responding organizations, 20 
used CBIMT. Of uses, 9 organizations 
used both CBT and CBIVT. None, 
however, used CBIMT to deliver more 
than 20% of their training. 

As shown in other North American 
studies, the largest organizations used 
CBIMT the most. However, over 35% of 
smaller organizations also used it 
(Table 3). 

CBIMT use varied as a function of 
industrial sector. Organizations in the 
goods producing, financial, and 
transportation, utility and communication 
sectors used CBIMT the most. Those in 
the retail and wholesale trade, government, 
and education services sectors used it the 
least (Table 4). 

What CBIMT Was Used to Teach 

Both Computer Based Training and 
Computer Based Interactive Video were 
used most to teach people how to use 
computer programs. As well as computer 
training CBT was used most to teach 
"hard" skills such as equipment operation 
and product knowledge, CBIVT, on the 
other hand, was also used to teach 
communication skills. That may be 
because video is often used to teach "soft" 
skills and video is a component of CBIVT. 



Who Was trained Using CBIMT 

In keeping with the differences in content 
taught, CBT was used most to train 
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professional staff, technical staff and 
office/administrative support while CBIVT 
was used to train technical staff, managers 
and supervisors. 

Equipment Used to Deliver CBIMT 

IBM and IBM compatible were the 
computers used most frequently to deliver 
CBIMT. Macintoshes and Amigas were 
used in a minority of cases. 

Advantag es and Disadvantages of CBIMT 

Flexibility and individualization were 
considered to b§ the greatest advantages of 
CBIMT and cost and availability of 
courseware the greatest disadvantages. 

Expectations of F^i^ir e Develop ment 
or Use 

Most developers and users expected to 
maintain or increase development and/or 
use of CBIMT. This suggests a high level 
of satisfaction. 
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1. To what extent does information 
about organizations represented 
among the ASHROD membership 
apply to all Edmonton employers? 
Considering the encouraging results 
of this study, a study of a more 
representative group of Edmonton 
employers would be interesting. 

2. Do companies that develop CBIMT 
for sale tend to be small, 
independent, and locally based 
companies everywhere? Can we 
expect larger multi-branch and even 
multinational companies to take 
over this market or do smaller 
innovative companies have inherent 
advantages? 

3. Will the development and use of 
CBIMT among ASHROD 
organizations and among Edmonton 
employers in general grow? 

4. How will the introduction of new 
multimedia technologies such as 
digital video affect the development 
and use of CBIMT? 



Amaig the 22 non-users, only 2 were 
expected to implement CBIMT in the 
coming year while 6 were expected to 
investigate it. 

SUMMARY 

Among ASHROD members, there is a 
substantial amount of both development . 
and use of CBIMT. One of the more 
interesting findings is that there are several 
small, independent, and locally based 
companies developing CBIMT for sale. 
The other is simply that there is this much 
involvement with CBIMT in Edmonton. 

The study suggests several further 
questions. 
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Table 1 



Development of CBIMT by Respondent Organizations 



CBIMT Development 


Count 


Count 
Percent 




7 

5 
32 

44 


15.9% 
11.4% 
72*7% 

100.0% 


Neither CBT nor CBIVT 





Table 2 

CBIMT Development as a Function of Number of Employee 



1 

Number of Employees 
in North America 


Development of 


CBIMT 


Total 
Cases 


CBT but 
not CBIVT 


both CBT 
and CBIVT 


Neither 
CBT nor 
CBIVT 


Over 1000 


4 


1 


19 


24 


500 - 999 






3 


3 


250 - 499 






2 


2 


100 - 249 


2 


1 


3 


6 


50 - 99 






1 


1 




1 


3 


3 


7 




7 


5 


31 


43 
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Table 3 

Use of CBIMT by Industry Group 



Level of 
Use 


4 

Hirschbuhl Study 


Technologies in 
Services 


4 

Ashrod Study 


Highest 


• Airlines 

• Data Processing 

• Banks 

• Insurance 

• Manufacturing 


• Communication 

• Finance & Insurance 


• Mann f^a rt-nri nrr r T3 a^ia... 

iiaiiuidLLunny oc Kesou] 

• Financial, Banking, 
Insurance & Real est. 

• Business Services 

• Trans. Utilities & Cor 


Medium 


• Telephone 

• Health 


* Transportation 

* Wholesale Trade 

* Business Services 


• Retail & Wholesale trs 

• Health & Medical 


Lowest 


• Accounting 


• Retail Trade 

• Real Estate 

• Accomod. & Food 


• Government 

■ Education Services 
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Percentage of U.S. Organizations Using CBT 
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MULTIMEDIA— A Makeover 

Anita Knisbacher MA., MS.-Department of Defense 

AbstractrHow do you slow down the technological aging process? How do 
you restore to vigor a program whose aging body has run out of steam? In short, 
how do you make it young and sexy again? When your program is 
instructionally sound but the body and face start to go, it is time for a makeover. 
Back in the 1980s, the Department of Defense put together a team to showcase 
the potential of interactive video technology in foreign language instruction. The 
effort lasted several years and the product is as popular among learners today as it 
was back then. But as technology has continued to' evolve, the older and 
sometimes proprietary hardware has begun to show its wrinkles. This paper will 
describe the restoration process and its results. 

Introduction 

In 1992 DoD contracted with Loral Western Development Labs to rework two foreign 
language interactive video (IAV) programs, in Spanish and Hebrew. This paper will concentrate on 
the Hebrew effort. 

The hardware in the original program consisted of a Zenith 248 AT, a Pioneer 2000 video 
disk player, and a proprietary random access audio device known as Instavox. The authoring tools 
were Courseware Design System (CDS) and Courseware Design Language (CDL), used for fine 
tuning. Track two of the video disk was used for the MCEA (More Carefully Enunciated Audio); 
the Instavox 14" "flappy" contained idiomatic expressions, culture notes and an advanced level total 
comprehension listening test. 

The audio quality of the Instavox left a lot to be desired, and its specialized cable made 
widespread dissemination of the courseware impossible. However, at the time, this was the only 
random access audio device that could fulfill our IAV/CAI development needs. 

The new system consists of a 486 multimedia PC platform containing a 19" multisync 
monitor, 660MB hard drive, CD-ROM drive, SVGA card, video overlay card, digital audio card, 
videodisc player, audio mixer, amplifier, speakers and mouse. A motherboard with a clock doubling 
chip was added later to accelerate system response. In addition, a 90MB Bernoulli drive to facilitate 
data transfers between networks and an HP Laserjet Illsi printer are provided for each of the two 
network locations. 

The software falls into two categories: Vendor-supplied and Loral-developed. The actual 
coding was done in Authorware Professional for Windows 1.0, with a later upgrade to the 2.0 
version. The Hadar Hebrew font selected for the conversion process did not project very well in 
certain point sizes until Loral modified it through the use of Fontographcr, an applications tool. 
Loral also developed other applications, such as a record keeping tool, which follows the learner's 
path through the various complex layers of courseware, monitoring and keeping track of progress. 
More about this later. 

The Learning Theory 
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Very early in the analysis phase of the original program, the Subject Matter Expert (SME) 
designer came upon a macrolevel strategy and structure for the development of instructional design 
by Reigeluth (1979, 1980, 1982, 1983) called Elaboration Theory. 

Consider Gagne's conditions of learning. Gagne (1984) advocates that instruction be based 
on prerequisite knowledge, and, as a result, concepts must be taught first, then rules and principles, 
followed by problem solving at the application level. Elaboration theory, by contrast, starts with 
the overall picture (without much detail), elaborating from there as needed. One may start with a 
very simple elaboration, epitomizing (presenting the essential features), then summarizing and 
synthesizing before going one level deeper, to the next elaboration. 

The strength of elaboration theory is precisely in this macrolevel approach. In most 
conventional courseware the learner is exposed to only a part of the overall structure at any given 
time. In elaboration theory the smaller parts are always related to the larger view, which procedure 
seems to increase the ability to learn concepts, promoting better transfer of information and 
enhancing long term retention. Like Norman's Web theory, Reigeluth's elaboration theory also 
holds that "memory is a network of ideas interrelated horizontally and vertically to the whole." 



The Course 

The product itself, an intermediate colloquial Hebrew language course, is based on a Laser 
disc containing two 30-minute situation comedies from the Israeli TV series Close Relatives. Side 
A, "The Maid," contains approximately 160 hours of instruction revolving around the controversial 
issue of hiring an Israeli Arab student to help with housework. 

Typically, the constraints of normal platform instruction make it very difficult to address 
different learning styles and degrees of readiness. However, the elaboration theory approach lends 
itself very well to the needs of a wide population of learners. 

After a thorough content analysis based on the fundamentals of elaboration theory 
(Jonassen, 1989), the overall structure was arranged in a top-down fashion. Principles, concepts and 
a few procedures were identified and sequenced accordingly. 

There are three distinct entry levels in the course: Film, Scene, and Clip levels, each 
capable of interrelating with the other or standing on its own. Film level covers the entire 30- 
minutes of video, together with related learning materials and overall testing. The Film level 
material was divided into seven contextual meaningful portions (Scenes) which constitute one 
layer of elaboration. Each scene is composed of about four minutes of video with related learning 
aids and appropriate tests. Finally, each Scene was divided further into as many contextual video 
bites (Clips) as possible. A typical scene might contain as many as five Clips or as few as two. 
These range from between thirty seconds to two minutes of video and are packed with valuable 
information, practice and remediation, putting the learner in complete control of path and pace. 
There are no tests at this level. 

Even though a few, strong learners may wish to enter the course at Film level and proceed 
very rapidly through it, most students will opt to view the entire video at Film level first, and then 
proceed down to the scene/clip levels, returning to the Film level once again at the end, to take the 
final examination. 

This three-level division, with the top layer at each level synthesizing the elaborations 
connected to it, is reminiscent of Reigeluth's "zoom lens in a camera" approach; i.e., we always 
begin with the wide angle view and zoom in very gradually, summarizing and synthesizing as we 
move along. Within each of the above levels, prerequisites precede more complex structures, 
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becoming the foundation for new learning and the glue for cumulative learning. The student is 
given ample opportunity for practice at each level. These elaborations become denser at the lowest 
level of instruction in order to build a strong foundation for the weaker learner. Throughout, the 
learner is able to control the mode and pace of instruction. 

At Scene and Film levels, there are test item banks, that allow the computer to scramble 
the items before each new attempt. All practice materials are geared to the tests. The learner is 
allowed only one try per each test item, with instant remediation provided for every wrong reply. 
And last but not least, while the clip level materials are geared to the slow or average learner, to 
challenge the stronger student, there is an optional advanced level total comprehension graded au*'al 
exercise in rapid-fire Hebrew. It is presented in dialog format between two speakers and raises the 
course level to 3+. 

"Mediated" learner control is the overall philosophy. The learner has full control over the 
sequence of learning, however, CMI (Computer Managed Instruction) steps in to guide and alert 
him/her (via a diagnostic examination) as to what segments or lessons need additional reviewing 
prior to taking the final test. This built-in "advisor", based on data collected by the record-keeping 
tool, sets the learner on the proper learning path, making the learning process more focused and 
efficient. 

With every item of the video sound track analyzable down to the individual utterance, with 
culture and idiom notes, with tutorials on almost every aspect of the grammar and with exercises to 
reinforce every point related to content, the program provides challenge and remediation for every 
type of learner, whi'e preparing all to pass the comprehensive Film-level final examination. 

Conversion Preliminaries and Course Strategy 

Since the design of the original course material is still valid, the conversion effort dealt 
primarily with hardware and screen design issues. The original DoD SME teamed up with the 
contractor to make sure that the change of face did not inadvertently modify the course design, and 
to assist with target language issues, such as quality assurance. 

A series of meetings between the Government and Loral representatives preceded the 
actual conversion process. By September of 1992 the parties came to the following terms 
regarding the implementation strategy (1992): 

® There would be a "maintainable interface" between the system (DOS/Windows) and the 

language enrichment materials; i.e., the government would be able, on its own, to later add 

to or modify the final product. 
® The original instructional materials would be preserved; in the new hardware setup the 

major apparent change would be the replacement of the Instavox proprietary 14" random 

access audio device with a Windows-standard audio card. 
® The original screen design would be modified and updated to give the product a complete 

facelift. 

® The text and graphics woold be similar to the original presentation, but updated to their 
new environment. 

® Screen prints from the original course would be used to transfer the content to its new 

environment, and for documentation purposes. 
® p 'udent registration into the courseware would be via floppy disk, making it possible for 

mc system to associate the user with data on his/her use of the enrichment materials. 
® Student paths through the course and test item data analysis and evaluation would be 

developed as a separate module, to facilitate its use on any Windows-based machine. 
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<E> Record keeping data would be written to floppy diskette. Only the enrichment materials 

would reside on the hard drive. 
® During the development phase, audio and video reference numbers, authoring models and 

screen printouts would ensure consistency of the final product. 
® That final product would also include a student guide and course administrator's handbook. 

The former would deal with al! information necessary for accessing the course, and the 

latter, with course registration and record-keeping. 
® Movement through the course would be via the keyboard and mouse. 

The Conversion Process 

The actual development phase of the conversion began when Loral started identifying audio 
and video frame numbers and putting together pre-determined Models (pre-programmed templates) 
representative of the various portions of the courseware. While the coder concentrated on coding 
issues, the graphic designers worked closely with the original course designer/SME to modify the 
outward appearance of the course; i.e., the presentation screens. 

The Government SME was on hand for consultation at all times, transcribing the cultural 
and idiom notes from Instavox, re-recording the transcribed notes and going over data files in order 
to eliminate typing errors before the materials were coded. 

The SME review of the final product began shortly after the course started going on line. 
A standardized mechanism known as Technical Problem Reports (TPRs) was developed for the 
SME to communicate problems and their required fixes. As with any software review process, there 
were, ultimately, hundreds of corrections to be made. In general, the policy was that functional 
inconsistencies had to be dealt with right away; textual content and typographic errors would be left 
until coding was complete. 

A weekly technical issues meeting was held between the Government SME and Loral's 
coder, minutes of which were presented at the general weekly meeting for TPR review and 
correction prioritization. 

The Converted Course 

The user interface allows learners full mobility among the various levels of instruction and 
from one activity to another no matter where they are. It is divided into four major areas: Level 
Selection area, Capability Selection area, Work area and Video/Audio Control area (Appendix 2). 

After the title screen (Appendix 1), learners encounter the videodisk selection screen. Once 
the disc side has been confirmed, they enter one of three levels of instruction: Film, Scene or Clip 
(via MENU). These are displayed on the user interface screen in the Level Selection area. Course 
Reference and HELP also reside here and can be accessed at all times (Appendix 2). 

The Capability Selection a is next (Appendix 2). It is preceded by the Title box which 
holds the address of the selection, reminding learners where they are. The Capability Selection area 
is equivalent to a menu, keeping the learners abreast of what selections are available. A muted icon 
is out of bounds for the duration of that activity. A selected icon is blue, indicating an activity - 
which is in progress By the same token, a blue-bordered work area indicates an ongoing activity. 

Next comes the Work Area. (Appendix 3). The open half of the screen has been divided 
into four quadrants. The bottom right has been allocated to video only. The other three quadrants 
alternate among the various selections (from the Capability Selection area). No two activities can 
share a common work (screen) space simultaneously. A dialog box will alert the user of the need to 
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close an open window prior to selecting another activity. Sub menu selections appear in the box 
blow the Capability Selection area as they become available. 

Finally, there is a Video Control area, located across the bottom of the screen (Appendix 
4). It contains a video locator bar whose color and appearance change, depending on the level the 
user is accessing. Its function, however, remains the same; i.e., to indicate what scene/clip is being 
viewed. The icons above the locator bar allow the learner the option of listening to an utterance 
over and over again, or playing a previous or next utterance at will. The Video/Audio Control area 
contains the VIDEO, SLOW AUDIO , AUDIO ONLY, AUDIO NOTES, and PAUSE icons. The 
AUDIO NOTES hold the entire content of the original INSTAVOX - idiomatic expressions, 
culture notes and the audio track for the aural Advanced Total Comprehension graded exercise. 

The only icon we have not mentioned is EXIT (Appendix 2). This icon is present no 
matter where in the course the user happens to be. 

First time users are taken straight to the tutorial (HELP, Appendix 2) prior to 
commencing instruction to familiarize them with all the intricacies of this complex design. The 
tutorial introduces the course through explanatory text, allowing the selection of various areas of 
the screen to learn more about them. After completing the tutorial, the learner can begin the 



course. 



The HELP (Appendix 2) icon allows the learner to review the tutorial or to display the 
Hebrew keyboard, leaving it on the screen as a reference for CLOZE and transcription exercises. 

MENU (Appendix 2) takes the user to a graphic representation of the course. By selecting 
the various click/touch areas on the title bar, the learner can see a summary of a particular scene, 
preview the video, or make Scene/Clip level selections (Appendix 5). 

At Film level the learner can access video, video with cultural and idiomatic expressions, 
word banks, a final examination, and the advanced-level listening comprehension graded exercise. 

Scene level provides all of the above plus various word reviews and a diagnostic and final 
(Scene level) examination. 

At Clip level the learner can have video, keywords, idiom notes, culture notes, a transcript 
of the text, an accompanying English translation, transcript-associated keywords, idioms, culture 
notes, specialized vocabulary notes and a three-level grammar review and practice. Every clip also 
contains comprehension exercises. 

Although each learning event or activity has a distinct window design, there are some 
common characteristics among the various windows: The title bar contains the title and topic of an 
activity from the Capability Selection area. It may also show the number of attempts per item, 
provide hints, directions, videodisk availability, plot, characters, English/Hebrew translations, 
grammar notes, exercises and various word review games. 

Conclusions 

The coding and pilot testing were completed in November of 1993. The course will be 
ready for distribution and field testing as soon as final revisions are made. 

The existing (old) version of the course has been in continuous use since 1988, and, even 
with the dated, clumsy interface, jerry-rigged hardware and clunky processor, continues to be highly 
popular with our students. If the updated version is even half as successful, we can expect many 
more years of useful life. Makeovers, of course, cannot work miracles; there has to be underlying 
substance to work on, in this case, solid instructional design. 
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Interactive Hypermedia For Training Electronic Surveillance Personnel 
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Abstract: The cognitive goal of naval electronic surveillance personnel is to match 
intercepted signal parameters with the names of specific emitters and their associated 
platforms. It was hypothesized that an instructional hypermedia-based training system 
will improve the electronic surveillance technician's ability to classify emitters. The 
training system provides opportunities for the technician to interact with a simulation 
of the electronic surveillance environment A graphic representation of the console of 
the electronic surveillance device was presented to the technician. The training 
procedure consisted of hooking an emitter symbol and selecting various multi-media 
representations of the emitter including sound, text, digitized photographs, and 
animated graphics. Emitter classification training was administered to 48 subjects 
using either instructional hypermedia or a printed listing. The training using the 
instructional hypermedia was found to be more effective than the training using the 
printed listings. 



Background and Problem 

On most US. Navy ships electronic warfare (EW) specialists operate electronic surveillance devices to detect 
and intercept radar signals. The cognitive goal of naval electronic surveillance personnel is to match intercepted 
signal parameters with the names of specific emitters (e.g., radars, missiles) and their associated platforms (e.g., 
ships, aircraft). The electronic surveillance fechnician searching for emissions must be trained as if the electronic 
surveillance devices had no automatic identification capability. The electronic surveillance technician must rely 
solely on intercepted signal parameters as the starting point for emitter classification and identification. 

Objective 

The objective of this project was to design, develop and test a training intervention to improve the electronic 
surveillance technician's ability to classify emitters. 

Instructional Hypermedia 

Although introducing technology into the training environment does not necessarily enhance learning (Clark, 
1983). incorporating hypermedia (Conklin, 1987) into an instructional delivery system may increase the 
effectiveness of Navy training. A hypermedia system is a computerized database consisting of text, graphic, 
video, and sound. The database consists of a system of nodes and links. Nodes store a quantity of information. 
Users navigate through the system of nodes via links. The hypermedia system creates a structuring of the data or 
a knowledge base. Hypermedia may create an effective learning environment because users actively associate 
knowledge when searching paths and retrieving information (Anderson, 1988). 

It is hypothesized that an instiuctionai hypermedia-based training system will improve the EW specialist's 
ability to classify emitters. The proposed training provides opportunities for the technician to interact with a 
simulation of the electronic surveillance environment. The training is self-paced and under learner control. A 
graphic representation of the electronic surveillance device console displaying numerous emitter symbols is 
presented to the technician. The training procedure consists of hooking a symbol and selecting various multi- 
media representations of the emitter including sound, text, digitized photographs, and animated graphics which 
enables the technician to link digital parameters to concrete images of radars, platforms, and weapon systems. 
The technician can hook more than one emitter so that representations can be compared and contrasted. This 



1 The opinions expressed in this paper are those of the author, are not official, and not necessarily reflect the 
views of the Navy Department. 
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approach will help the tcchician identify emitters and their respective platforms because interactions with the 
instructional hypermedia will create elaborate cognitive structures (Reigeluth & Stein, 1983) of the emitter data 
bases. 

A computer-based lesson incorporating the instructional hypermedia approach was developed for a database 
of 35 emitters. The instructional hypermedia was tested in two phases. Experiment I tested for immediate recall 
of the to-be-leamed material and Experiment II tested for recall of the to-be-learned material immediately and two 
days after the intervention, and included a cooperative learning treatment group. 





The emitter is a SPS-67 surface 

arch radar, a solid state G/H/ 
band system designed to replace 
the SPS-10. The system is in full 
production for the U.S., German 
and Australian navies. 



BRG: 085 
FRQ: 99999 
PRF: 99999 STD 
SCN: 99999 MS 



The guided missle cruiser Home (CG-30) is in the Belknap class of cruisers. 
It has a displacement of 7,930 tons when it is fully loaded. It is 547 feet long, 
carries 477 personnel, and cruises at about 33 knots. 



Figure 1. An Instructional Hypermedia Display 



Method 

Treatment Group 

Treatment group subjects received a computer-based emitter classification lesson incorporating the 
instructional hypermedia. The lesson consists of two parts: Frceplay and Drill. The Freeplay displays the 
electronic surveillance deviccs's console to the user. An example of emitters displayed on the "wagon wheel" of 
the electronic surveillance device during Frceplay is sketched in figure 1. During Frceplay, the subject clicks on 
emitter symbols using a mouse and then selects sound, picture, animated graphic, and textual representations of 
the emitter. During Drill, the user practices associating emitter parameters with radars and platforms. For the 
emitter parameters displayed, the user selects the correct radar and platform. Feedback is provided for every 
response. The treatment lasted about 25 minutes. 
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Control Group 

Control group subjects received an alternative emitter classification lesson that resembles training that might 
be provided prior to a warfare exercise. Subjects were required to learn the emitter data base from a printed 
listing. The data are in notebook form and resembles an Electronic Order of Battle. The control training lasted 
about IS minutes. 

The subjects in both groups were administered a test prior to and after the intervention. The test contains 64 
multiple choice items that are presented randomly; 35 items require the subject to identify the correct radar when 
given the emitter parameters; 29 items require the subject to identify the correct platform when given the radar. 

Experiment I. This experiment consisted of two sessions, two weeks apart During the first session, the 
subjects were administered the multiple choice test During the second session, the subjects received the 
hypermedia or control training, immediately followed by the multiple choice test 

Experiment II. This experiment consisted of two sessions, two days apart During the first session, the 
subjects v ere administered the multiple choice test, the hypermedia or control training, and the multiple choice 
test again. During the second session, the subjects received the multiple choice test 

During experiment I, most of the subjects during the hypermedia training made specific comments about the 
emitters and associated multimedia representations to the experimenter. They wanted to provide some extraneous 
information about the displayed emitters. For example, "I saw that radar on a destroyer last year in the Persian 
Gulf' and "That radar is very similar to the radar aboard my ship" were said to the experimenter. This reaction to 
the hypermedia should have been expected because much of the training for electronic surveillance devices is 
provided in the operational environment (i.e., on the job). In this environment the EW technician must 
successfully interact with other surveillance personnel as well as the other combat information center personnel. 
Consequently, the hypermedia training intervention was tested in a cooperative learning setting (i.e., a cooperative 
learning treatment group) in experiment II. It is believed that subjects working together will recall and synthesize 
more information than they could working alone (Webb, 1990). 

Subjects in the cooperative learning group were individually pre-tested and then randomly subdivided into 
pairs. The hypermedia training was administered to the pairs, but the subjects received the post-tests individually. 
It was predicted that the treatment group and the cooperative learning treatment group would significantly 
outperform the control group on the post-intervention tests. 

Results 

Twenty subjects were tested for experiment I and 28 were tested for experiment II. For experiment I, data 
was analyzed for 10 treatment subjects and 7 control subjects. Two experiment I subjects did not return for the 
second session and one experiment I subject's data were lost due to a computer malfunction. For experiment II, 
data was analyzed for 8 treatment subjects, 9 cooperative learning subjects, and 8 control subjects. Three 
experiment I subjects did not return for the second session. 

Experiment I. Mean test score by training group is shown in figure 2.. After training, the treatment group's 
score increased about 18% (f(9) = 6.39, p < .001) while the control group's score increased about 7% (/(6) =11.21, 
p < .001). The treatment group's performance gain was significantly higher than the control group's performance 
gain (F(l,14)= 15.73, p< .001). 

Experiment n. Mean test score by training group is shown in Figure 3. Immediately after training, the 
treatment group's score increased about 20% (f(7) = 6.86, p < .001), cooperative learning increased about 13% 
0(8) = 4.61, p < .01), and the control group's score increased about 7% (r(7) = 3.l0,p < .05). The treatment 
group's immediate performance gain was significantly higher than the control group's immediate performance gain 
(F(l,13) = 13.20,p < .01). Although the cooperative learning group's immediate performance gain was higher 
than the control group's immediate performance gain, the difference was not statistically significant 
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Each group's performance declined little two days after training. When compared to the pretest score, the 
treatment group's score increased about 17% (r(7) = 8.04, p < .001), cooperative learning group's score increased 
about 14% (/(8) b 4.33, p < .01), and the control group's score increased about 6% (/(7) = 2.44, p < .05). Again, 
the treatment group's performance gain was significantly higher than the control group's performance gain 
(F(l,13) ~ 1 1.49, p < .01). The cooperative learning groups performance gain was higher than the control group's 
performance gain, but this difference was not statistically significant. 

Discussion 

In this study training was developed to improve the EW specialist's ability to classify emitters. Two training 
modules were developed: One using an instructional hypermedia approach (the treatment) and the other using a 
traditional approach (the control). Subjects were trained using instructional hypermedia in pairs and alone. 
Subjects were tested prior to, immediately after, and two days after training. Technician performance improved 
significantly for both hypermedia and control groups even two days after training. The hypermedia was effective 
for training subjects in pairs or individually. 

It was hypothesized that emitter classification training employing instructional hypermedia would be more 
effective than training using a printed listing. Training using instructional hypermedia was found to be 
significantly more effective than training using print only when provided to individuals working alone . Results 
using instructional hypermedia in a cooperative setting were promising, but need to be to supported with future 
research. Previous investigations of computerized instruction in academic settings have found that performance 
improved when students were paired (Dalton, 1990; Stephenson, 1992). Cooperative pairs of students were better 
able to reorganize and clarify, guide and correct, and build on each other's ideas. Students working in cooperative 
learning environments have been found to have reduced anxiety, an increased willingness to ask for help, and an 
increased willingness to take feedback seriously than students working alone. A closer examination of the effects 
of instructional hypermedia in cooperative settings is needed. 

Hypermedia technology is becoming widespread and is a suitable candidate for delivering emitter 
classification training. In response to post-training questions, all subjects in the treatment groups thought that the 
instructional hypermedia helped their performance and should be implemented. The instructional hypermedia was 
found to be easier to use (t(32) = 2.1 1, p < .05) and more enjoyable (t(30) = 3.43, p < .002) than the control 
training. Moreover, when given a choice of training, all control subjects who were shown the treatment (after the 
post-test) preferred the instructional hypermedia over the control training. 

Our investigation provides evidence for using instructional hypermedia for training electronic surveillance 
personnel to classify emitters. In addition to validating an instructional technology approach, a considerable step 
was made toward designing a practical system for training. Therefore, with appropriate attention to 
implementation, this computer-based instructional system will impact naval electronic surveillance. 
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Abstract 

This paper presents the results of an experimental evaluation of the Mobile Subscriber 
Remote-telephone Terminal (MSRT) Tutor (the MSRT is a mobile phone that the Army 
uses, but whose operation is much more difficult than the commercial equivalent). The 
evaluation was conducted in the regular training environment used for communications 
training in the U.S. Army. Soldiers who were required to know how to operate the 
MSRT, but had not yet learned, were used as participants in this study. Two treatments 
were compared. The computer-based tutor was "pitted" against a treatment that had the 
participants operate the actual equipment. We assumed this "hands-on" approach would 
be superior to the computer tutor. The test was to perform the operations procedures on 
the actual equipment. Accuracy and time were measured. Results indicate that both 
groups are equally accurate, but the hands-on group was able to perform the procedures 
more quickly. We conclude by recommending the computer-based tutor because of the 
prohibitive costs associated with hands-on training. 



The first Intelligent Tutoring System (ITS) was built over twenty years ago (Carbonell, 1970). 
Since then, much work has been done in the area designing more "intelligent" systems by improving 
student, expert, and pedagogical modeling (see, for example, the special issue on student modeling in the 
Journal of Artificial Intelligence in Education (McCalla & Greer, 1993) for a series of articles on this 
advancement). However, very little effort has been done in the area of evaluating the effectiveness of ITS. 
In fact, one of the major criticisms of this field of inquiry is the lack of evaluation data (Rosenberg, 1987). 
Some have suggested plans for evaluation (Littman & Soloway, 1988; Shute & Regian, 1993), while 
others have begun the process of conducting evaluation (Anderson, Boyle, & Reiser, 1985; Corbett & 
Anderson, 1991; Shute & Gawlick-Grendell, 1993; Shute & Glaser, 1990). The results of this evidence are 
that estimates of effectiveness based on Bloom's (1984) 2-sigma problem place ITS at about a 1.0 effect 
size (Legree & Gillis, 1991; Legree, Gillis, & Orey, 1993). However, this estimate is based on a limited 
number of studies that have some limitations. The intent of this paper is to add to this evidence by 
reporting the results of a field-based evaluation. 

Background 

Legree and Gillis (1991) have done an extensive review of evaluations that have been conducted. 
They distinguish between extensive and minimal evaluation. Those evaluations that have had a long 
enough treatment have produced the 1 .0 effect size. Those that have not been large scale systems have not 
produced the 1.0 effect size. Of the studies that they reviewed that had the 1.0 effect size, one is difficult to 
fully understand (Anderson, et al„ 1985). Anderson, et al.'s, study has been cited frequently for its impact 
of tutoring, yet it does not include details of the experiment (it was published in a popular computing 
magazine, not an experimental professional journal). That leaves only a very few extensive, well-designed 
studies that are used to account for the effect size of ITS. 

Shute and Regian (1993) intimate that much research is underway at Armstrong Labs related to the 
effectiveness of ITS systems. These studies are well-controlled experimental studies that use "paid" 
participants to learn from their systems. These experiments are controlled very well and have great internal 
validity. However, these experiments have little or no external validity. In fact, Farquhar (1993) present 
the results of an experiment with a tutor that trains people to operate a loading crane through a simulated 
control panel. This crane and its simulated control panel do not exist. It is a tutor for experimental 
purposes only. These results need to be verified with real learners learning real things in the real world. 

Shute and Regian refer to these real world examples as quasi-experimental studies because often 
they do not use random sampling and there is very little experimental control. They characterize a well- 
controlled experimental design that takes place in a real world setting with real learners who really need to 
know the content as "Research Utopia". This "Utopia" is the goal of the current effort. 
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Method 

Shute and Regian specify seven factors in designing a "good" evaluation. They are: 

1 . Delineate the goals of the tutor. 

2. Define the goals of the evaluation study. 

3. Select the appropriate design to meet the defined goals. 

4. Instantiate the design with appropriate measures, number and type of subjects, and control conditions. 

5. Make careful logistical preparations for conducting the study. 

6. Pilot test the tutor and the study, and 

7. Determine the primary data analyses as you plan the study (p. 249). 

I will use these as a framework for explaining the methodology of this experiment. 
Tutor Goals 

First, you need to define the goals of the tutor. In this case, the tutor is designed to teach a learner 
how to operate a piece of Army communications equipment called the MSRT. It consists of three primary 
components. The first is something that is very much like a telephone except that it has programming 
elements for security purposes. This component is common in the Army communication network. The 
second component is the radio that has a variety of switches, knobs, buttons and lights. The last part of the 
equipment is the power supply that has a variety of switches. The goal of the tutor is to teach soldiers how 
to power up the equipment, load security information, link into the communication network, and finally, 
place a call. For a complete description of the MSRT Tutor, see Orey, Trent and Young (1993). 

Study Goals 

Second, you need to define the goals of the study. I want to know which is better — the MSRT 
Tutor or the best available training (tutoring while operating the actual equipment). The MSRT Tutoring 
environment is a hypermedia simulation of the MSRT. I believe that it is better to train on the actual 
equipment. Unfortunately, to operate the MSRT, $5 million worth of equipment and eight switch 
operators need to be "on-line" in order to use the MSRT. In addition to the equipment, you still need to 
have someone or something provide the instruction. I chose to use tutors to guide the learner as they used 
the MSRT equipment. Although this is costly, I wanted to compare the MSRT Tutor to the best available 
instruction. 

Experimental Design 

Third, choose an experimental design to meet your goals. Using Shute and Regian's (1993) terms, 
I chose to use a summative, benchmark, "quasi-experimental" design. However, they sometimes use 
"quasi-experimental" to mean real-world and other times it means to use random sampling. Both of these 
are important experimental issues. The basic design takes place in the real world, but I attempt to use 
experimental principles. I begin by using a program in LinkWay Live! to randomly sample two groups. 
The design of this program is to create equivalent size groups. That is to say, the program asks for the size 
of the pool of subjects. It runs once by using the random number generator to place each person into one 
of two groups. At the end, if the size of the two groups is not equal (or off by one in the case of an odd 
number of subjects in the pool), it runs again until the groups are equal. The program prints a list of E's 
and Cs (for experimental and control) which is then used in conjunction with an alphabetical list of 
participants for the assignments. 

Design, Measures, and Subjects 

Their fourth principle involves experimental logistics (appropriate measures, controls, sample 
sizes). I decided that the goal of the tutor was to teach someone to operate an MSRT, so the test ought to 
have the learner operate an MSRT. I use trained observers, a check list of steps, and a stop watch to 
measure their performance. The check list is used to generate a measure of accuracy while the stop watch is 
used to measure their speed. In order to have consistency between observers, I video taped a group of 
soldiers as they performed operation procedures on the MSRT equipment. These video tapes were edited 
into two tapes - one for practice and one for calculating inter-rater reliabilities for the observers. These 

125 



133 




collection took eight hours. 

The entire pool of participants from a class was used in the experiment. The experiment used 53 
officers. Five allied officers were dismissed from the experiment because they had limited English fluency. 
The remaining 48 were randomly placed into one of two groups. This group of officers was new to the 
Army. This was their first assignment. The MSRT Tutor was designed to teach general users. The 
population that I had access to was Signal Corp soldiers. Although a Signal Corp soldier does not have 
exactly the same background as other general purpose users, many characteristics arc shared. They all were 
required to know how to use the MSRT. The experiment used officers who had just entered the Army, 
therefore, there background knowledge of communcations equipment would be about the same as the general 
purpose user population. 

Because the existing instruction included a one-hour lecture on a variety of aspects of the MSRT, I 
chose to have both groups receive this instruction. Further, both groups viewed a training video on MSRT 
operation prior to working with the separate treatments. Finally, both groups gathered around a single 
MSRT as its operation was demonstrated by the instructor. Each group received these common 
instructional treatments on the morning of their respective treatments. The same instructor and materials 
were used on both days. After these pre-treatments, the Computer group was turned loose on the Tutor and 
the Hands-on group came to the MSRTs in groups of six (one MSRT per learner). The "tutor" then guided 
the learners through the operation of the MSRT. In the case of the Hands-on group, once they had 
performed the operations procedures, they sat down and the next group came forward. Frequently, others 
would gather around the MSRTs and look on while a group of six was practicing the procedures. After all 
the groups had their hands-on experience, some came back to practice again. Once all were ready, the 
MSRTs were separated from the learners with a partition and small groups were "tcstcd-out" on the 
procedures (I used 5 raters for the experiment ). 

The Computer group went through the tutor twice. The operations procedures had been broken 
down into six problems. The tutor used a gross measure for mastery (fewer than 4 errors and 4 coach 
assists). After mastering a problem, the tutor went on to the next. At the end, the students were asked to 
do the tutor again so that they "mastered" all six procedures twice. As soon as they completed the tutor, 
they could "test-out" on the actual MSRT. 

The "test-out" had the learner come to the MSRT equipment with an observer standing next to the 
equipment observing the actions taken. A 2-page check list was created to be used as a guide to the 
observation. There were 47 separate steps on the check list. In the observer training, we decided which 
steps were considered potentially damaging to the equipment or were vital to operating the equipment and 
which could be left undone. There were 9 steps that could be left undone and the remaining ones were 
considered vital. If intervention was warranted, the observers were instructed to only state a vital step that 
appeared on the check list. No other assistance was provided. The observer could place a check (the learner 
successfully performed a step), a circle (the learner was told to perform a vital step), or a blank (the learner 
did not perform a non-critical step) next to each item in the check list. A composite score was created from 
this data. Providing assistance was deemed more damaging than not doing something (circles versus 
blanks). Therefore, we took the total number of steps and subtracted twice the number of circles and once 
the number of blanks, which is expressed like - (47 - (2X# of Cirlces)-# of Blanks). This figure 
was the accuracy figure. Speed was calculated in seconds. 

The learner was asked to perform the procedures twice immediately following the learning phase of 
the experiment. The rationale for this was that the Computer group would necessarily need to transfer their 
knowledge from what became known as the "cartoon" version to the real equipment. I felt that these 
mappings could be made in the first attempt and then their performance ought to be the same as the hands- 
on group. 

There were three side issues that were of concern in this effort. The first was to determine some 
more qualitative feedback about the learner's feelings about the instruction. A questionnaire with 12 items 
covering issues such as good use of graphics, overall value of the instruction, would you recommend to 
your peers, was developed for the experiment. In addition, I decided to explore the motivational aspects of 
the training using Keller's (1987) ARCS model of motivation. We used an instrument created by Chanlin 
(1993). Finally, I were interested in the differential effects of the treatment on decay. Therefore, I had the 
participants perform the operations procedures four weeks after the learning treatments. 



Logistics 



The fifth principle involves logistical preparations. This is truly the most difficult part of the 
"real world" experiment and the results show that this was somewhat limited. I randomly generated groups 
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for the experiment and sent them to the instructional supervisor, but the subjects were placed into one of 
two groups based on the upper and lower half of an alphabetical list. The two groups were to receive the 
same lecture portion, but the instructor's materials had been mis-sequenced. As a result, he cut the 
Computer group's lecture in half. Therefore, the groups were not exactly randomly sampled and the groups 
did not receive exactly equivalent treatments. The result was that the design became "quasi" experimental. 
Perhaps Shute and Regian (1993) were correct when they characterized field research as quasi-experimental. 

Pilot Test 

Pilot test the study is the sixth principle. A pilot test was run and many experimental problems 
were resolved. This was vital, and perhaps two or more pilots ought to be conducted given our experience. 
The pilot test resulted in modifications to the Tutor as well as modifications to experimental procedures. 

Data Analysis 

Finally, Shute and Regian (1993) suggest that you plan the data analysis procedures. There are 
essentially five separate analyses that needed to be done in this experiment. The first was already discussed 
- inter-rater reliabilities. A simple correlation matrix for the accuracy measure calculated from the values 
on the observation check list was used for inter-rater reliability. Because all evaluatcrs evaluate trie exact 
same person from the exact same perspective, this is a fairly accurate representation of the rating 
performance. In an earlier pre-pilot, we found that time was highly correlated, so we did not measure this. 
The results of the first experiment indicated that the evaluators needed further training that resulted in fairly 
consistent results in this experiment (See, Figures 1). 
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Figure 1. Inter-rater reliabilities for this experiment. 



The second and third analysis procedure is essentially the same. This is the primary analysis for 
this experiment. These two analyses are the analyses of the two primarily dependent measures - time and 
accuracy of performance. These measures are collected three times for each of the two groups. Therefore, a 
2X3 ANOVA with three repeated measures was used. I was interested primarily in the main effect related to 
grouping, but I also had an interest in the main effect for repeated measures in as much as I were interested 
in the decay factor. I also had an interest in any interactions because it would imply that the groups' 
behavior differed at different data collection times. Post hoc tests would be t-tests on each individual 
interaction element and an ANOVA for the main effect of repeated measure. 

The questionnaire was nominal data, so I used a chi-squared analysis to compare the groups. There 
were 12 items, so there were 12 separate chi-squared tests. The motivation instrument collapsed to four 
categories — attention, relevance, confidence, and satisfaction. These composite scores were compared using 
a t-tcst for each (total of 4). An alpha level of 0.05 was used on all analyses. 

Results and Discussion 

We begin with the accuracy measures of performance. The results of the repeated measures 
ANOVA indicated that there was a main effect overall for the repeated measure (F=65. 16,p<0.001). In the 
post hoc ANOVA using Fisher's PLSD, each time we measured their accuracy was different from each 
other. The best performance was the second time immediately following training ( X=42.7) followed by 
the lest immediately following training (X =37.2) and the worst was four weeks later ( J? =23.5). Of 
particular interest to this discussion is the fact that there is this large decay in performance. In later 
versions of the tutor, this aspect will be explored. The fact that there were no main effects for the 
experimental groupings (F=0.48,p=0.49) was indeed heartening. After all, we were attempting to see how 
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well the tutor stood up against the best possible alternative training. The fact that there were no 
interactions also was heartening (F=0.66, p=0.52). Refer to Figures 2 and 3 for details of this test. 



Means Table for Perfs 
Effect: Category for Perfs 
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Figure 2. Descriptive statistics for each accuracy of performance measure. 
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Figure 3. Interaction graph for the accuracy of performance measure. 

The results of the repeated measures analysis for speed were less favorable for the tutor. There was 
a main effect for group (F=7.01, p=0.01). Referring to Figure 4, clearly the computer group took longer to 
perform the tasks than the hands-on group. In fact, if you examine the graph in Figure 5, you will noiice 
that much of this difference was on the first and last tests (though there was no interaction, F=1.69, 
p=0. 19). I had assumed that the speed of performance had become essentially equivalent after the second 
immediate test. However, the speed of performance of the computer group seemed to have slowed to the 
same performance speed as their initial transfer task speed. Perhaps, one could argue that they were again 
relying on a computer-based representation in memory and were attempting to make mappings again to the 
actual equipment. However, further data would need to be collected to verify this conjecture. It should be 
noted that these same results were obtained in the pilot (although the computer group was faster on the 
decay test than the first immediate test). 
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Means Table for T-time 
Effect: Group 
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Figure 4. Descriptive statistics for time on performance. 
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Figure 5. Interaction graph for the time for performance measure. 

It would be nice to conclude that the computer-based tutoring was just as effective as the hands-on 
tutor instruction. Unfortunately, that is not possible. Although accuracy seems to be the same for both 
groups, speed seems to differ. Among many possibilities^ I think that two are of particular interest. The 
first idea is that the nature of the memory stores are different for the two groups. It would be interesting to 
perform some interviews with the subjects in each group to attempt to tease out the nature of their 
representations. A second idea for further study could be that the computer-based training "mastery" 
learning implementation is not robust enough. In fact, 1 initially put mastery determination in the hands of 
the learners and they could select which problem to work on. I found that many did not even do all the 
problems once. Therefore, I implemented an easy to implement strategy. A more "intelligent" strategy 
might optimize their practice and produce real mastery that would result in speed improvements. 

I also examined the learners' reactions to their instruction with a questionnaire. There were no 
differences between the two groups. Finally, we used Keller's (1987) ARCS model to determine if there 
were differential motivational effects. Again, it was found that there was no significant difference between 
the groups. This wouM support the idea that the computer tutor was the same as the hands-on instruction. 



Conclusions 



I was interested in comparing a computer-based-hypermedia-simulation-lTS with the best 
instruction 1 could devise. The results are somewhat confusing. While learners in both groups were 
comparable in terms of accuracy of performance, opinions about the quality of the instruction, and opinions 
about how motivating each of the instructional treatments were, there were differences in how fast they were 
able to perform procedures. 1 believe that this difference in performance can be attributable to the fact that 
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the computer group was required to transfer their learning from the computer world to the real world, 
something not required of the hands-on group. 

If each of the 47 steps in the operation of the MSRT required a mapping step for the computer 
group, then this difference would show up in their speed of performance. Further, the mapping is a process, 
not a memory. The knowledge of the step to carry out is a memory. The accuracy measure in the 
instrumentation indicated that there was no difference between the groups. They had similar "amounts" of 
memory. The speed was different. Therefore, one could argue that it is this mapping of domains that 
causes the difference in speeds. 

An alternative explanation would be that the computer group was in the declarative phase of skill 
acquisition and the hands-on group was at a procedural stage of learning (Anderson, 1982). We have a 
difficult time with this interpretation. One problem is that the hands-on group essentially performed the 
procedures once. According to Anderson, this acquisition occurs over a period of time. The computer 
group averaged 19 problems completed during the learning phase of the experiment (the tasks were broken 
down into six procedures, so they did varying numbers of these six procedures, but did on average 19 
problems in total). Therefore, deferring to Anderson does not make as much sense. 

The bottom line is that taken as a whole, hands-on instruction is better than computer-based 
instruction as conceived in this experiment. Therefore, one ought to conclude that the Army ought to use 
hands-on training for ail instruction. This is not completely true. The first problem is that in order for one 
MSRT to call another, $5 million worth of equipment needs to be operating. In addition, 8 people need to 
be manning thai equipment. This is quite costly. Another consideration is that the $5 million worth of 
equipment is also going to be used to provide hands-on training for its operators. So it may be likely that 
you have your people ready to perform the hands-on activity and it is not possible because the switching 
shelter is not operating correctly. I hesitate to say, that I never could make a call on the MSRTs used for 
testing. The experimenters explained how the equipment would have worked if it was operating correctly. 
Fortunately, these aspects are only at the very end of the procedures effecting the last 10 steps or so. Other 
equipment may not be so designed. 

The MSRT Tutor always worked. It always generated the correct sounds. It always displayed the 
correct fights. It was much more consistent than the actual equipment. Further, you need to pay someone 
to stand by and provide the hands-on training. With the Tutor, this is unnecessary. Therefore, it would 
seem that the Tutor is more cost effective than the hands-on training. However, I would not be honest if I 
did not admit that the hands-on experience was vital and should occur at some point, but the Tutor can be 
most valuable in the initial training phases. 

I would also like to conclude with the comment that we need more research (we would be out of a 
job if we didn't). There are a variety of things that can be done to minimize transfer problems. I already 
have a multimedia version of the Tutor that uses digital movies to explain tasks to perform. These movies 
could make these mapping and the representation be more like the actual equipment. Unfortunately, these 
movies take up much disk drive space (1 megabyte versus 19). There are cheaper solutions that may be just 
as effective. For example, digital still images could be just as effective as digital movies and because there 
are only three main pieces of equipment in the MSRT, only three digital pictures would be required. This 
would not take much storage. It does, however, require at least a VGA monitor. The MSRT Tutor was 
implemented in EGA graphics because of the huge number of EGA capable machines in the Army. These 
are not capable of displaying the photo realistic images. 

Other materials and adjunct teaching strategies might improve the Tutors performance including 
job aids and strategy instruction. These can have an immediate impact on performance and potentially can 
begin to address the decay problem. 
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Abstract: This paper describes research on simulator sickness and 
driver performance in the Iowa Driving Simulator* The four 
independent variables were driver experience, driver gender, width 
of the visual field, and presence or absence of motion. The primary 
dependent variables were symptoms of simulator sickness, driver 
ratings of the simulator's realism, and driving performance. 
Predictions for driver experience were confirmed, that novice drivers 
would experience less simulator sickness and rate realism higher 
than experienced drivers. Predictions for gender were confirmed, 
that females would experience more simulator sickness than males, 
but would rate realism the same as males. Predictions for visual 
field were not confirmed, that a wider visual field would result in 
greater simulator sickness and greater realism ratings. Predictions 
for motion were not confirmed, that the presence of motion would 
result in more simulator sickness and greater realism ratings. 

The Iowa Driving Simulator 

The Iowa Driving Simulator, or IDS, is a project of the Center for Computer Aided 
Design (CCAD) of the College of Engineering at the University of Iowa (Haug, 1990). 
Although the simulators have been designed and constructed by the College of Engineering, 
research with them is conducted by research groups from several departments including 
medicine, computer science, psychology, education, and engineering. 

There are three vehicle simulators at CCAD. The high fidelity simulator includes a 
motion base with 6 degrees of freedom, 191 degrees of visual field in the front and 64 
degrees in the rear, and realistic audio. Different model vehicle cabs may be mounted inside 
the simulator "sphere", although to date the main vehicles installed and simulated have been a 
Ford Taurus, a Saturn sedan, and a military HMMWV (pronounced Humm-Vee, and 
standing for High Mobility Multipurpose Wheeled Vehicle), The simulator can run with 
different scenarios, that is different roadways, scenery, etc., and during each run extensive 
data is collected on all actions taken by the driver. 

There is also a medium fidelity simulator with fixed-base, that is no simulation of 
motion, and an increased visual resolution capability. This can also run with different model 
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vehicles. It is capable of the same scenarios and data collection as the high-fidelity 
simulator. 

Lastly, there is a low fidelity and substantially lower cost simulator without motion, 
with lower fidelity visuals, limited scenarios, and restricted data collection capabilities. 

The University of Iowa has received a large grant to design and build the National 
Advanced Driving Simulator, scheduled for completion in several years. This is expected to 
include a more sophisticated linear motion base and 360 degrees of visuals. 

Research projects in progress at the center include human performance research, traffic 
safety research, driver health research, vehicle engineering research, and basic simulator 
design research. 

Our own work concerns human performance research, including the use of driving 
simulators for training and for assessment. The research reported here deals with the causes 
of simulator sickness and how that and other variables affect user performance in driving 
simulators. Simulator sickness is a phenomenon akin to regular motion sickness, although 
the causes, symptoms, and persons susceptible to it are often different (Reason & Brand, 
1975; Reason, 1976; Kennedy, Dutton, Ricard, & Frank, 1984; Frank &Casali, 1986). 

This study utilized the high fidelity simulator with the Ford Taurus. The research 
participant, i.e. the driver, and the experimenter, who sat in the front passenger seat, enter 
the simulator by a staircase. The simulator consists of a sphere mounted on the hydraulic 
motion base. Video projectors inside the sphere (3 for the front view and 1 for the rear 
view) project animated scenery, roadways, and other moving vehicles on the inside walls of 
the sphere, which act as the video screen. There are also several speakers for realistic audio. 
The rear screen permits the driver to use the side-view and rear-view mirrors normally. The 
driver can look directly to either side and still see realistic images. 

The motion base, when used, is programmed in synchrony with the visuals, so that as 
the driver turns, stops, or hits a bump, the visuals reflect the motion and the motion base 
attempts to recreate the correct feel. 

Although the Iowa Driving Simulator is technologically very impressive, many people 
question (and understandably so) the value of a multi-million dollar driving simulator. It is 
one thing to spend a few million dollars on a simulator for a passenger airliner which itself 
costs 50 to 100 million dollars, or a military fighter jet which costs even more. But an 
automobile only costs 15 or 20 thousand dollars. Why spend several million to simulate it? 
There are several reasons. They include the capacity for exact replication of events, 
extensive and detailed data collection, safety while teaching or assessing dangerous 
conditions or maneuvers, the capacity for artificial guidance and feedback, increased 
efficiency by compressing the time-frame, and the ability to do either part-task or full-task 
training. 

But there are other impediments, besides the considerable cost, for driving simulator 
use in training and assessment. Our two main concerns are simulator sickness and validity 
(Whiteside, 1983). Simulator sickness, which occurs in about 20 percent of the Center's 
younger (that is under 60 years old) and 33 percent of the older research participants, varies 
from slight sensations of lightheadedness to extreme nausea. The existence of simulator 
sickness decreases the validity a simulator. That is, it decreases the confidence we can have 
that a person's performance in the simulator is the same as, or predicts, what their 
performance would be like in a real vehicle. For that reason, our main interest is to 
determine the factors which most influence the occurrence of simulator sickness and devise 
methods for eliminating it as much as possible. Of course, simulator sickness is just one 
phenomenon which affects the validity of a simulator. In general it is believed that simulator 
fidelity, or realism, is the main factor affecting validity (Gagne, 1954). 

Experimental Design and Procedure 

The study we have just completed with support from the Link Foundation investigated 
the effects of four factors on simulator sickness and driving performance in the simulator. 
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The factors were driver experience, driver gender, width of the visual field, and presence or 
absence of motion. Our subjects were either junior high school students with no driving 
experience or adults with many years of driving experience. All subjects drove the same 
high-fidelity simulator, but some of them had the motion base turned on and some had it 
turned off. Similarly, some subjects had a complete 190 degree visual field (in front) and 
some had a more narrow 60 degree visual field. Experimental design was four factors 
completely crossed 

Participants drove the simulator one at a time, always accompanied by the 
experimenter. Before driving the simulator, participants took a few short tests to assess their 
sense of balance and other feelings of physical state which may be affected by simulator 
sickness. They answered a questionnaire about driving experience and if they had any 
history of ordinary motion sickness. 

They were then taken into the vehicle simulator. The experimenter always sat in the 
front passenger seat and directed the driver as to when to start, turn left or right, stop, and so 
on. The roadway consisted of a rural portion, a freeway portion, and an in-town portion. It 
took about 40 minutes to complete. While driving, the experimenter observed and took 
notes on any symptoms of simulator sickness and wrote comments on important aspects of 
driving performance, such as if the driver went off the road. 

After driving, the participant was given a short test and questionnaire to assess any 
symptoms of simulator sickness, and to get the participants opinion of the realism of the 
simulator. 

Hypotheses 

Our main hypotheses were that non-drivers would experience less simulator sickness 
than drivers, that females would experience more simulator sickness than males, that a wider 
field of view would produce more sickness, and that motion would produce more sickness. 

We also expected the following differences for perceived realism - that experienced 
drivers would be able to differentiate realistic and unrealistic conditions better than non- 
drivers, that the wider visual field and presence of motion would increase perception of 
realism, but that there would be no difference in perceived realism for females versus males. 

Results 

Our findings confirmed the hypotheses for both experience and gender. Non-drivers 
experienced much less simulator sickness, irregardless of the other factors. Non-drivers 
also rated realism higher than experienced drivers, the latter being rather critical of the 
simulator's realism. 

Experienced drivers had more or less simulator sickness depending on the other 
factors. Female drivers had considerably greater incidence of simulator sickness and more 
females dropped out of the study because of simulator sickness. Yet there was no difference 
in perceived realism for males versus females. 

Our hypotheses concerning the two fidelity variables, visual field and motion, were not 
confirmed. The width of visual field affected neither degree of simulator sickness nor the 
level of perceived realism. 

Nor did the use of motion have the affect we predicted. However, there was a trend 
opposite our prediction for motion and perceived realism. Subjects using the simulator with 
motion rated the realism lower than those using the simulator without motion. 

What affect did our factors, or the occurrence of simulator sickness, have on driving 
performance? We have only begun to analyze that data. Each participant's 40 minute 
driving session produces approximately six million data points. Our continuing analysis is 
focusing on two questions. What impact do our factors (gender, experience, visuals, 
motion) have on performance, and what impact does the occurrence of simulator sickness in 
drivers have on their performance? 
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Discussion 

Our next step it to investigate in more detail those people who do suffer simulator 
sickness. This should allow more sensitive investigation into its causes and possible ways 
to control it Future use of the simulator for either research or training depends on being able 
to reduce simulator sickness, either through practice, scenario design, fidelity adjustment, or 
simulator improvement We p^n to pretest participants in a short run to determine who will 
experience simulator sickness. We will ask those participants to return for experiments in 
which we vary conditions more carefully (such as the type of motion) or try methods 
intended to reduce their incidence or severity of simulator sickness, such as pre-training or 
starting out on a straight roadway with few turns. 

How do our results relate f o the current literature? Simulator research to date does not 
indicate any gender differences, but that is because almost all simulator research has been 
with men in predominately military type vehicles planes, helicopters, tanks, and 
submarines (Kennedy, Dutton, Ricard, & Frank, 1984). The potential use of simulators for 
ordinary ground vehicles (cars, trucks, buses) is much greater, with equal numbers of men 
and women as operators. It is essential that research and development with driving 
simulators consider the variable of gender. 

Concerning fidelity variables, previous simulation research and literature reviews, 
which have rarely done direct experimental comparison of fidelity variables, as well as being 
based largely on military devices used by men, has suggested that increasing field of view or 
adding motion will increase simulator sickness (Casali & Wierwille, 1986). Our results have 
not confirmed that, especially concerning visual field of view. 

Previous research has not dealt much with the experience variable. The finding that 
non-drivers experience little simulator sickness and rate simulators relatively higher in 
realism, has positive implications for simulator design and use. Simulators for training and 
assessment of novices may be quite adequate even with low levels of fidelity, with 
concomitantly lower cost and possibly less incidence of simulator sickness. 

Determining the causes of and ways to control simulator sickness are an essential step 
in creating simulators which are valid for research and training. For example, there is great 
interest in the use of the simulator for medical research investigating the effects of aging, 
medication, eye impairments, and the like on driving. Obviously the use of a simulator for 
such research is much safer and therefor more ethical. But if participants experience 
extensive simulator sickness, how are we to know if impaired driving performance is the 
result of a medication or the result of simulator sickness? 

We believe that simulator fidelity is one of the important determinants of simulator 
sickness. But we are interested in simulator fidelity for other reasons. Higher fidelity drives 
up the cost of a simulator. If particular tasks, such as a medical research experiment or a 
state driving test, can be done with a lower fidelity simulator and still be valid, it will be 
much cheaper. For a long time it was believed that higher fidelity is better and leads to 
greater validity (Gagne, 1954; Miller, 1974; Hays, 1980). Years of research with flight 
simulators has cast doubt on that. Our intention is to conduct research on the affect of 
varying fidelity on validity for different puiposes, such as for instruction versus assessment. 
It has long been contended that greater fidelity is needed for a test than for instruction 
(Gagne, 1954). Research in visual impairments may well require high visual fidelity but 
allow low kinesthetic fidelity, that is, may not require the use of an expensive motion base. 

One of our goals is to try to control simulator sickness and at the same time optimize 
instructional effectiveness through the use of gradually increasing fidelity (Alessi, 1988). 
This might be done be starting instruction in a low fidelity simulator, which is cheaper, and 
moving to higher fidelity simulators as the learner accommodates and as performance 
improves. It may also be done with dynamic fidelity, increasing and decreasing realism as a 
function of the learner's performance and well-being. However, this technique requires a 
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high-fidelity simulator even for lower fidelity conditions, so would not benefit from as great 
a cost savings. 

Our finding that non-drivers experience little simulator sickness and rate the simulator 
high in realism lends credence to the use of low fidelity simulators for teaching and assessing 
driving performance with beginners. It may not even be necessary to gradually increase 
fidelity with non-drivers, but to go direcdy from low fidelity simulators to real vehicles. 

The next step in this program of research, and which we are already beginning with 
data from the current simulator sickness study, is to determine the best methods of assessing 
driving performance in the simulator. There are different ways of approaching this, from 
very atomistic analyses of driver actions, such as steering wheel control and lane deviation, 
to task analysis of the more global aspects of driving performance, such as maintaining safe 
speed, passing, turning, and stopping. 

Obviously there are a lot of variables and considerations in the use of driving 
simulators for both research and training purposes. At a minimum, the problems inherent in 
simulator sickness must be overcome to gain the advantages engineers are fond of claiming 
for their simulators* 
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Abstract: Although Problem-based Learning (PBL) has been adopted as a 
method of instruction for many health education institutions world-wide, it still has 
many opponents. The major objections to PBL are: (1) PBL is an inefficient method 
of instruction since it requires students to gather information through self-directed 
learning, (2) PBL is perceived as costly since it requires a greater investment of 
faculty time to function as tutors, and (3) PBL is more difficult and costly in terms of 
evaluation of student learning. Furthermore, there is a lack of direct evidence that 
all the goals of PBL are being met. This paper discusses a modification of the PBL 
format and the use of computers to alleviate these difficulties associated with 
problem-based learning. 

As medical knowledge has grown so has the need to improve the methods of instruction employed in 
training individuals employed in the healing arts. Problem-based learning is an instructional format 
which has been developed in an attempt to meet the need for improved instruction in health-related 
sciences. According to Barrows (cf. Norman & Schmidt, 1992), the goals of PBL are: (1) to promote 
self-directed, life-long learning, (2) to foster clinical reasoning or problem-solving skills in students, (3) 
to enhance acquisition and retention of knowledge by situating learning in the context of its real-world 
application, (4) to enable students to better integrate basic science knowledge into the solution of clinical 
problems, and (5) to increase student intrinsic motivation. 



Barrows (1983) provided a detailed explanation of the problem-based learning format. PBL is 
implemented in small groups of from 5 to 6 students and a faculty tutor. At the beginning of the week 
the tutor gives the students a clinical problem which they are to attempt to solve without receiving an 
explanation of the problem solution. After presentation of the problem the students attempt to identify 
the nature of the problem and to generate as many hypothesis as possible. As the students discuss the 
case, they are encourage to seek information relevant to the patient's history, physical examination, and 
diagnostic tests. This information is provided by the tutor who has access to the printed case book for 
each problem. Once the students have gathered information from the case book they choose from their 
original list of hypothesis what they believe are the mechanisms causing the patient's condition. At this 
point the group, with the assistance of the tutor, decides what learning issues are involved in the case, 
decide how to go about resolving these issues, and delegate among themselves assignments for resolving 
these learning issues. Each student may take a different topic and use various learning resources in 
researching his or her assigned learning issue. After a pre-determined length of time (usually 2 to 4 
days) the group reconvenes and is again guided by the tutor through an examination of the same case. In 
this instance, however, the students act as experts for the particular topic they were assigned. When the 
students have finished a case they critique their performance and summarize what they have learned and 
how it relates to what they have learned in the past. In the whole process, the tutor is never to act as a 
dispenser of knowledge, but rather as a facilitator of the interaction among the students. Other faculty 
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members may be assigned as resource persons that may be consulted by the students but they are not 
allowed to solve the case for the students. Formative evaluation for the students in performed by the 
tutors and students as they engage in the PBL methodology, Summative evaluation occurs periodically 
via the means of standardized simulations administered by technical personnel. There are variations to 
the PBL format just described. Other schools vary the size of the group, method of presentation, the 
amounts and timing of feedback, and the means of evaluation. However, this paper will discuss PBL in 
terms of the curriculum described by Barrows. 

PBL Strengths and Weaknesses 

The Goals of PBL 

Although problem-based learning is generally well-accepted by the schools and students using this 
mode of instruction, many questions remain as to whether or not the goals of PBL are being 
accomplished. Research has been conducted that has revealed several weaknesses in the problem-based 
learning curriculum. One study (Patel, Groen & Norman, 1991) examined the effects of conventional 
and problem-based medical curricula on problem solving. Although students in the PBL curriculum were 
able to provide more detailed explanations of the basic science concepts involved in the cases presented 
to them, they also displayed a tendency to generate many more errors than the conventional curriculum 
students. Patel and colleagues argued that the errors committed by the PBL students were probably due 
to the scarcity of feedback given by their tutors as well as to confusion resulting from the intermingling 
of instruction in basic and clinical sciences in the PBL cun^ulum. They also found that the problem- 
solving method taught to PBL students may interfere with the normal problem-solving nvnhodologies of 
expert clinicians. 

Other studies question the ability of instruction to teach general problem-solving skills. In his review 
of research on problem solving, Norman (1988) noted that the main difference between expert clinicians 
and medical students is that clinicians generated better hypotheses, not that they were better problem 
solvers. Norman also found that an expert is an expert because he has ready access to the domain- 
specific knowledge which enables him to recognize the pattern of the problem before him. 

In their review of research on problem-based learning, Norman and Schmidt (1992) found that (1) 
there was no evidence the PBL improved general problem-solving skills; (2) PBL tends to reduce initial 
levels of learning but improved long-term retention of knowledge; (3) PBL may enhance transfer of 
concepts to new problems; (4) PBL tends to enhance intrinsic motivation in students; and (5) PBL seems 
to enhance self-directed learning with this tendency being retained by the learner. 

Inefficiency of PBL 

Patel, Groen, and Norman (1991) found that the students in a conventional medical curriculum 
learned to solve clinical problems more efficiently than PBL students. In problem-based learning, studems 
are not typically given a solution to the problems they are asked to examine. Instead they are asked to 
discover the problem solution through independent research, group discussion, and tutor guidance. 
Needham and Begg (1991) conducted a series of experiments comparing problem-oriented training and 
memory- oriented training in their ability to promote spontaneous analogical transfer. As a part of their 
research, Needham and Begg gave one group of students instruction on how to solve a given problem and 
asked them to remember the problem solution so they could solve additional problems. A second group 
of students were given the same problem and asked to attempt to solve the problem before the problem 
solution was given to them. However the second group of students were not asked to remember how to 
solve the problem. Both groups were then given problems to solve which were analytical to the first 
problem. It was found that the group who was asked to attempt to solve the problem before attempting a 
problem solution were more successful at transferring the problem solution to an analogous problem than 
was the memory-oriented group. Needham and Begg also discovered that experimenter-provided 
explanation of the problem solution was necessary if transfer was to occur. It appears from these 
findings that PBL could be improved, both in terms of efficiency and in promoting spontaneous 
analogical transfer of problem solutions, by allowing tutors to administer appropriately timed feedback 
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concerning problem solutions. 

Administrative Cost of PBL 

As far as the cost of the problem-based learning curriculum, Barrows (1983) found that while the 
initial costs of implementing PBL were higher, normal operating costs were the same as those of 
conventional curricula. Barrow's assessment of costs, however, may not be applicable in schools that 
operate at substantially lower budgeted levels, such as colleges of veterinary medicine. The fixed costs 
associated with PBL may fit within the budgets of medical schools but be beyond the means of other 
institutions of health care education. Regardless of the afTordability of the PBL curriculum, finding a 
way to lower the cost of its administration would not be a wasted effort. 

Evaluation 

Barrows (1993) describes the benefits of using simulated patients as part of a standardized simulation 
for student assessment. While this method of student evaluation could be assimilated with minor 
modifications to other fields of health education, the cost of this assessment technique would exceed the 
cost of administering evaluation in a conventional curriculum by a significant amount. Alternate methods 
of student assessment need to be explored in order to find evaluation strategies comparable to using 
simulated patients but at a reduced expenditure of time and money. 

Using Computer Simulations in a Modified PBL Curriculum 

The preceding evaluation of the strengths and weaknesses of problem-based learning suggests the 
need for a modification of the PBL curriculum to amend problems associated with its lack of efficiency 
as well as the utilization of computers to improve its economy. The following scenario is one possible 
means of employing these changes. In this method of instruction, the students will meet initially in small 
groups. The groups will use a computer to access a previously assigned prototypical problem. The 
computer will allow students access to history, physical examination, and diagnostic data for the case 
being presented. At various stages of the program the students will be directed to identify learning issues 
associated with each problem and enter them into the computer. As the various groups work through the 
problem, their progress may be monitored at any time by the instructor via a local area network or 
collected and evaluated at a later date. From the list of learning issues, each student will be assigned a 
research topic. The research will be conducted in one day rather than in two to four days however. 
When the students reconvene they will reexamine the example problem and attempt a solution, again 
using the computer and their progress being monitored as before. Next all the groups will meet together 
with the instructor. Regardless of whether or not the students reached a proper problem solution, the 
instructor will provide an expert description of the problem solution and give each group written 
feedback concerning their efforts. As a part of this process, the students will receive instruction in the 
basic science concepts associated with the problem solution. This procedure will be repeated for a group 
of prototypical problems designed to cover a unit of instruction. After the students have completed a unit 
of instruction individual students will use the computer to practice problems analogous to the problems 
presented in the unit. As the students attempt to solve the analogous problems, the computer will again 
track the method by which the attempts are made and record this information in a database record for the 
student. The computer will compare the student's problem-solving methodology to a standardized 
assessment form and record the differences. The next step will be to give the student feedback as to how 
his attempt at the problem solution may vary from the standardized format. According to Norman and 
Schmidt (1992), it is important for immediate feedback to be given to students after they have attempted 
to solve a problem. Finally, at the end of each unit of instruction, the students will be evaluated on the 
basis of their performance on a standardized simulation administered by computer. 

While the preceding instructional format is as yet untested in its final form, it should retain the 
advantages of the problem-based learning curriculum while compensating for its major deficiencies. By 
interjecting the instructors explanation of the problem solution and instruction in the basic sciences 
associated with each problem solution, greater instructional efficiency should be enjoyed. This strategy 
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should also serve to promote transfer of problem solutions to analogous problems. Use of the computer 
in presenting problems, monitoring student progress, and administering assessment should provide PBL 
with greater economy. Although the outlook appears promising, further research is needed to assess the 
potential benefits of this instructional strategy and its proposed method of implementation. 
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Abstract: The aim of the project Problem Analysis with Computer Organized 
Simulations (PACOS) is to solve two problems a course on unsteady flow analysis 
suffers from: students are unfamiliar with problem analysis and students start 
studying too late. PACOS was used for the first time in November 1993. The 
results are very promising. Students that use PACOS get familiar with problem 
analysis and receive higher exam scores than students that do not use PACOS. A 
large number of students start their study early (during the first half of the course), 
especially the good students. Furthermore, the PACOS activities of good students 
appear to be qualitatively different from those of poor students. 

As participants of the project Problem Analysis with Computer Organized Simulations 
(PACOS) the authors have created a new user-interface which was used successfully at 
the end of 1993. The project concerned the effective use of simulations in education, 
including educational research. It was sponsored additionally by Delft University of 
Technology. 

The old course and its problems 

The course in question consists of three elements: 

- Lectures concerning tidal wave propagation, flood waves in rivers, harbour 
oscillations, etc. Over a period of 7 weeks, 2-hour lectures are presented twice a 
week. 

- A syllabus offering the theoretical foundation of these phenomena. 

- Three assignments in which students individually analyze representative phenomena. 
The students complete the course by taking an exam. 

In the past these three elements were challenging enough to prepare students for the 
written examination successfully. In recent years, however, the more open examination 
questions regarding design, prediction and interpretation gave insufficient results. These 
might have arisen because of the two following problems. First, students are not familiar 
with problem analysis and therefore are not skilled in answering these types of questions. 
Second, a large number of students are not prepared sufficiently to take the exam. These 
students start studying too late, although they are invited (from the first week) to study 
the physical aspects of wave propagation including storage, inertia, friction, reflection, 
resonance etc. 
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The philosophy of the new course 

The new course was intended to overcome the two problems mentioned above by 
stimulating the self-instruction regarding mathematical and physical aspects of the 
phenomena during the first two weeks of the course and by basing this self-instruction 
upon problem analysis. Students appear not to be stimulated by lectures- and assignments 
only. The new course offers an extra stimulus by using a new instructional format called 
Problem Analysis with Computer Organized Simulations (PACOS). Students learn to use 
three analytical models more effectively and to apply numerical calculations by means of 
a non linear simulation model called DUFLOW. 

According to this idea, the structure of the course in the first three weeks was 
changed: earlier explanation and application of analytical solutions, the first of the three 
assignments in a shorter period of time, and more attention to predictions by means of 
simple analytical solutions. In the fourth week, three days were available at which each of 
a maximum of 200 students had access to a computer for two hours of using PACOS. 

In the new course students study the physical aspects of a tidal irrigation system 
(ERRSYS). Attention is given to the Harmonic Method, applied to a network of 18 
sections. Furthermore, the student learns to use a 'simple analytical model' (SAM) for 
stating predictions concerning ERRSYS during the problem analysis procedure. The 
predictions based on SAM, which contains only one section and a discrete storage area, 
are stated within a few minutes. Furthermore, they seem to correspond well to the results 
of excessive calculations by DUFLOW, based on a non-linear model that contains 18 
sections. (See figure 1). 
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Figure 1. The two models of IRRSYS as used by the three calculation methods 

Before starting PACOS students have to take a test called PACO. Most questions 
contained in PACO are aimed at changes in IRRSYS and solved by applying the 
Harmonic Method. Students that fail the test are not permitted to use PACOS. 
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PACOS 



PACOS is a new user-interface buik upon the hear: o: a j.a:;on package. Tnis user- 
interface supports the student in four ways: 

- it presents assignments to the student, in coherence with :he steps in an analysis 
stracegy; 

- it forces the student to state a prediction before running the simulation; 
it simplifies the complexity of the input of the simulation package; 

- it presents questions in order to interpret simulation results. 

Prior to starting PACOS, students are invited to prepare some parameter variations 
individually and use SAM to state predictions of the tidal difference and the amplitude of 
the discharge. In PACOS each student is presented 10 assignments . In the first 6 
assignments the student is asked to think of a variation of ERRSYS by changing one 
parameier in one or mere sections. The sradent is free to choose any one of the available 
parameters and :s free to decide upon the magnitude of the change ! 'oniy very extreme 
changes are not acceptec In :he rwo following assignment :wo parameters can be 
changec simultaneous:*/ The rwc .as: assignments are of i different nature: instead of 
analyzing cr.anges in ERRSYS. :r.e sruaen: has :c conserve: i variation of :he network 
system :r.at re acnes a cenam goa_. An exampie of sucn i go a. :s" :r.e veiociry in the tidal 
inie: has :c oecorr.e smaller man 3.5 m/s 




Figure 2, An example of a screen containing output and an interpretation question 



The first eight assignments are related to an experimental cycle well known in theories 
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of scientific reasoning (Khlar & Dunbar, 1988) and have been studied in educational 
research (de Jong & Njoo, 1990; Reimann, 1989): 

1. Analyze the situation. 

2. Generate a hypothesis. 

3. Test the hypothesis. 

3a. Design an experiment and state a prediction. 
3b. Run the experiment and interpret the output data. 

4. Evaluate the results. 

In PACOS students have to state a realistic prediction before they are allowed to run the 
simulation. They can use SAM to evaluate the prediction before running the simulation. 
Students learn to use predictions in order to get insight in unsteady flow analysis as well 
as to prevent useless experiments and an overload of data. The educational relevance of 
stating predictions and confronting these with observed outcomes has been emphasized 
elsewhere (Friedler, Nachmias & Linn, 1990; Reimann, 1992). 

The use of DUFLOW, a tool for practitioners and researchers, is rather complex for 
students. It will take an inexperienced user at least a couple of hours to get familiar with 
DUFLOW and another couple of hours to complete some assignments. If that person is 
not familiar with computers, it might even take days. The PACOS user-interface is that 
simple that even students who are not famiUar with computers are able to finish the as- 
signments within the time limit of one and a half hour and concentrating on problem 
analysis only. 

The instructor can use a simple file editor to create questions which are presented 
along with the graphs prepared by PACOS (see figure 2). These general questions assist 
the student in interpreting the output by directing his attention towards interesting aspects 
of the graphs. These graphs illustrate the relevant physical aspects of wave propagation. 

Method of evaluation 

The course in question is scheduled in the third of four years of the academic study 
for Civil Engineer. The course started in November 1993, and the exam was scheduled in 
January 1994. A total of about 145 students that entered university from 1990 onwards 
enrolled in this course. 

The procedure was as follows: 

- A pretest was given during the lectures at the end of the third week. 

- The computer-based instruction was scheduled in the fourth week. Students worked 
with a computer for 2 hours, in which both PACO (0.5 hour) and PACOS (1.5 hours) 
had to be finished. Students that failed PACO were not allowed to start PACOS. 

- A posttest was given during the lectures at the end of the fourth week. 

- A questionnaire was handed out during the lectures at the end of the fifth week. 
Both the pretest and the posttest contained questions related to problem analysis. 

Results 

The results of the various tests were mixed. On the one hand, the pretest-scores (taken 
by 67 students) as well as the posttest-scores (taken by 57 students) were bad. These tests 
were passed by 11 and 7 students respectively. All of the 7 students that passed the 
posttest did use PACOS. On the other hand, the PACO test-scores were good. A total of 
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122 students took the test, 113 of whom passed it. 

There was no complete overlap between PACOS-users and students who took the 
exam. Of the total group of 145 students: 

- 101 used PACOS, 28 of whom did NOT take the exam, 

- 109 took the exam, 36 of whom did NOT use PACOS, 

- 8 did neither. 

On the exam, the students that did use PACOS outperformed the students that did not 
use PACOS (t=2.13; p=.018). This difference is also found for the PACO-test (t=2.43; 
p«.013) and the posttest (t=2.15; p=.020), but not for the pretest (t«-.47, p=.319). 
These findings seem to suggest that students that did use PACOS did start studying in the 
fourth week (after taking the pretest but before taking the PACO-test) and were better 
prepared to take their exam. 

Of the students that participated in PACOS, a number of 59 students completed 
assignments 1 to 8 successfully. Assignment 9 was accomplished successfully by 33 
students, the more difficult assignment 10 by only 12 students. 

The predictive power of PACOS 

For each student the computer kept a record during the use of PACOS. Among other 
things, this record contains for the first eight assignments the number of simulation runs 
(N)> the number of different variables that were changed (Var), and the average value of 
the relative magnitudes of the changes (dVar). Furthermore, three performance measures 
were calculated: 

- The difference between predicted and calculated tidal difference (dG). The measure 
ranges from 0 (good) to approx. 1 (poor). 

- The difference between predicted and calculated amplitude of discharge (dQ), ranging 
from 0 (good) to approx. 100 (bad). 

- The absolute value of a comparison of ratios (of predicted and simulated values) of G 
and Q. This measure tests the coherence of G- and Q-predictions related to the 
physical properties of the system and to SAM, even in case the dG and dQ measures 
are not very accurate (dG/dQ). The measure ranges from 1 (good) to 0 (poor). 



Table 1 

The correlation scores between the various measures 





dG/dQ 


dG 


dQ 


pretest 


posttest 


dG 


-.53 










dQ 


-.37 


.29 








pretest 


-.12 


-.09 


.22 






posttest 


.30 


-.11 


-.09 


n.a. 




exam 


-.05 


-.04 


.01 


.12 


.52 



Because the dG/dQ measure took into account the coherence between the dG and dQ 
measures, it was expected to be the Best predictor of the exam-scores. However, table 1 
shows that the exam-scores correlate moderately positively with posttest-scores, but not 
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with the other measures. The dG/dQ, dG and dQ measures correlate a little with the 
posttest-scores, and, not surprisingly, with each other. These findings suggest that at least 
two elements contribute to posttest performance. Since the posttest is based partly upon 
insight into IRRSYS, one element is related to PACOS. The other element is related to 
the exam (knowledge of and insight into unsteady flow analysis). The weak correlation 
between PACOS-measures and exam-scores might be accounted for by lack of transfer 
(students are not able to apply their insight into IRRSYS to other tidal systems). 

The differences between good and poor students 

One point of interest is the way in which good performers differ from poor 
performers. Two types of differences can be distinguished: differences in the way they do 
PACOS, and differences in their opinion on PACOS. 

The figures in table 2 make clear that good students on the dG/dQ, Dg and dQ 
measures, compared to poor students, execute more simulation runs (N) and take more 
variables into consideration (Var). Furthermore, students that are good at predicting dQ 
use more reasonable variable changes (dVar). Thus, in general, the activities of good 
students during the PACOS session appear to be qualitatively different from those of poor 
students. Compared to poor students, good students analyze IRRSYS both more deeply 
(they start mere simulation runs) and more broadly (they change more different 
variables), and the magnitudes of the changes they made were smaller (more realistic).' 



Table 2 

The differences on N, Var and dVar between the 30% best and the 
30% worst performers on PACOS-measures 





dG/dQ 


dG 


dQ 




best 


worst 


t 


best 


worst 


t 


best 


worst 


t 


N 


5.20 


3.88 


2.14* 


5.46 


3.76 


2.81** 


5.08 


4.00 


1.75* 


Var 


4.52 


3.44 


2.32** 


4.83 


3.40 


3.16** 


4.60 


3.36 


2.78** 


dVar 


1.69 


1.98 


0,56 


1.88 


1.80 


0.15 


1.35 


2.34 


1.86* 



*p < .05; **p < .01 



In order to compare good and poor performers on their opinion on PACOS, a measure 
is needed to distinguish good from poor performers. One of three measures could be 
used: dG/dQ, dG or dQ. Since this measure correlates the best with the other PACOS- 
measures and with the posttest-scores (see table 1), the dG/dQ measure is used to make 
this distinction. 

Opinions on PACOS were collected by means of a questionnaire in which students had 
to rate 27 items on a 5-point scale (l~low, 5=high). 58 students completed the 
questionnaire. When comparing good and poor students according the dG/dQ measure, 7 
items show significant differences (see table 3). From the first two lines in table 3 it is 
clear that, compared to poor students, good students see a clear link between PACO and 
the preceding lectures and they indicate to have spent more time on preparation. Thus, 
spending more time on preparation has a positive effect on performance. 
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Furthermore, the last 5 lines seem to indicate that poor students have a more favorable 
opinion of PACOS. One point of interest is the fact that poor students are very positive 
about the interpretation questions that the lecturer has added to PACOS in order to 
confront students at the right moment with the important physical aspects in the graphs of 
PACOS. 

Table 3 



Significant differences in answers from poor and good students 





mean value 
poor std*s 
in — So) 


mean value 
good std's 
\n — jo) 


t 


P 


PACO links up 
wiui lectures 


3.00 


3.53 


2.01 


0.026 


Time spent on 
preparation 


2.90 


3.67 


1.87 


0.034 


Simulations are 
useful 


4.43 


3.87 


2.08 


0.025 


Simulations are 
motivating 


4.05 


3.40 


1.80 


0.042 


Too many aspects 
are repeated 


2.19 


2.73 


1.93 


0.031 


The interpretation- 
questions are useful 


4. 10 


2.93 


3.78 


0.001 


User-instructions 
are sufficient 


3.90 


2.67 


2.03 


0.027 



Discussion 

The course in question suffered from two problems: students are not familiar with 
problem analysis and students start studying too late. The use of new computer 
simulations, scheduled early in the course, was thought to offer a solution to these 
problems. The main purpose of this study is to evaluate whether this is true. Another 
purpose is to assess differences between good and poor students. 

Concerning the problem of being unfamiliar with problem analysis, it is clear that a 
large number of students have improved their problem analysis skills by actual 
experience. A total of 101 students did use PACOS, 59 of whom completed the first eight 
assignments successfully. The 9th and 10th assignment were completed successfully by 
only a small number of students (33 and 12 respectively). These figures indicate that, 
halfway the course, students can handle the type of assignment related to the experimental 
cycle (the first 8 assignments) and that they have difficulties with the constructive type of 
assignment (assignments 9 and 10). Since the constructive assignments were scheduled at 
the end of the computer session, the bad results could be caused by time pressure. 
The improvement of problem analysis skills is also shown by the fact that students who 
used PACOS received higher exam scores than students who did NOT use PACOS. 
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Finally, nearly all students appear to appreciate PACOS and find the PACOS simulations 
both useful and motivating. 

Concerning the problem of a late start, 45% of the students that completed the 
questionnaire indicate to have spent at least 4 hours of extra self-instruction on analytical 
solutions during the first half of the course. The fruitfulness of this extra self-instruction 
is reflected by the fairly high pass rate of the PACO-test, which amounts to 
approximately 90% of the students involved. 

When compared to poor students, good students on the dG/dQ measure analyze 
IRRSYS both more deeply and more broadly* and the magnitudes of the changes they 
make are more realistic. Since they also indicate to have spent more time on extra self- 
instruction, spending more time on preparation has a positive effect on academic 
performance. 

To conclude, the findings in this study suggest the following. 

- Students that use PACOS get familiar with problem analysis and receive higher exam 
scores than students that do not use PACOS. 

- A large number of students start their study early (during the first half of the course), 
especially the good students. 

- The PACOS activities of good students appear to be qualitatively different from those 
of poor students. 

However, from this study it cannot be concluded that early preparation will improve the 
ability to analyze problems: it might be that the good students just have more effective 
study habits or strategies. Furthermore, it cannot be concluded that PACOS improves the 
scores on tests related to problem analysis, at the moment the posttest was taken. The low 
posttest-scores might indicate that students still have difficulties to answer that type of 
questions easily within a short time and without the possibility of gaining credit-points. 
However, the students that pass the posttest are likely to be the best students at the exam. 
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Abstract: Machine learning approaches can assist human productivity in using 
automated tools. There are numerous computer programs that help professionals 
and scholars complement their higher-order thinking skills, but such programs 
are complex and require so much effort to learn that users may spend more time 
on computer problems than they do on their domain tasks. Computer-aided 
instruction (CA1) often does not address the learner's problem and may be too 
time consuming. Hypertext information systems and electronic performance 
support offer means by which knowledgeable users can locate necessary 
assistance, but they must be aware of their own needs. Our research is focused 
upon machine learning which can individualize and prioritize the interaction 
between the user and the computer, adapting to the needs of users through 
pattern matching and other intelligent >nethodologies. Potentially, machine 
learning systems will automatically acquire knowledge from human-computer 
interaction in real world situations. This paper presents a methodology for an 
integrated broad-based research project. 



Introduction 

Technology has irreversibly changed our lives. Many of us pursued a teaching career, which 
meant something much different a few years ago. From the lime of Socrates, the pendulum has swung back 
and forth between different philosophies of education. Does better understanding come about through 
questioning or memorization? Is experience more crucial than reading? Is a liberal education better than 
specialization? But the present shift is more than a pendulum swing. A few years ago, hypermedia and 
micro-worlds were unheard of. Accessing information meant visiting a large library with a good cataloging 
system. Education with and for technology has added new dimensions to every educational controversy 
and has made us educational technologists. 

When Arnold and Huxley debated educational issues over a century ago, the outcome may not 
have had life-threatening implications. Now we are told that the world is different because of information 
technologies and that schools must prepare citizens for this different world. It has been estimated that, "A 
weekday edition of the New York Times contains more information than the average person was likely to 
come across in a lifetime in seventeenth-century England" (Wurman, 1989). Our educational systems arc 
expected to teach students how to integrate vast quantities of information into their learning, thinking, and 
finally into their jobs. If present day education doesn't teach students how to benefit from information 
technology, they will be limited to unskilled positions (Rumberger, 1987) - and a lower quality of life. 

Computer Assisted Higher-Order Thinking 

If the real pay-off of technology is to be realized, more and more professionals and scholars will 
use tools that will automate processes once considered too time consuming or unpredictable to be easily 
accomplished by humans. Technology Can reduce some occupations to lower level skills; for instance, bar 
code scanners make the checker's job a matter of physical manipulation rather than memory and association 
tasks. However, the desired result is that people will be able to develop greater creativity and flexibility 
with the help of computers. The computer will foster thinking by functioning as a "cognition enhancer" 
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which "combines the complementary strengths of a person and an information technology" and can 
"enhance human accomplishment by a division of labor: the machine handles the routine mechanics of a 
task, while the person is immersed in higher order meanings" (Dede, 1989). The result, much to be desired, 
is empowering the human mind to generate new solutions to complex problems or productive thinking 
(Mayer, 1983). 

Learning to use computers in the manner described must become a natural process. In the climate 
of nurturing higher-order mental processes, education will never be complete. As new technology is 
developed, both young people in school and adults on-the-job will continue to study and learn. Schools and 
workplaces will become centers for extending human capacities through partnering with automated 
processes. And the partnering will be extended to include other human beings through telecommunications 
technology that will permit collaboration on a scale never before possible. These technologies arc 
extremely complex, and as the complexity of the systems increases the difficulty of making productive use 
of them increases. Many professionals report spending more time mastering their computer systems than 
they do on their specialties. The task of using automation must be simplified even though the programs 
themselves and the domains they address can only be expected to become more complex. 

Automated Tools 

Automated tools are widely used. Professionals who practice musical composition, medical 
diagnosis, genetic research, financial decision making, and building design can all make use of 
computerized systems that aim to improve productivity and quality of results. Our research has been aimed 
at architects and engineers who may use many types of automated systems for design. Computer Aided 
Design (CAD) is the most commonly used automated tool in the design environment. The most widely 
used CAD systems can produce the detailed documents necessary for planning, review, construction and 
management of a facility, all higher-order mental processes. CAD systems have existed for over two 
decades and are still used mostly as drafting tools, rather than design tools, by many building designers 
(Liker, Fleischer, & Arnsdorf, 1993). However, two large surveys of government architects and engineers 
indicated that they found CAD helpful in the design process and that it allowed them to consider different 
possibilities in design (Shaw & Golish, 1988, and Shaw, Golish, Webster, & Yang, 1991). Frequently 
designers respond to the question, "Is CAD a good design tool?" with the thoughtful explanation, "No, but I 
can use it." 

CAD systems are complex and not easily supportive of concept design, but the indications are that 
any effort to improve designers' understanding of CAD systems would be generally useful and perhaps 
conducive to the design use of CAD. That potential plus their wide use has warranted that our research be 
conducted with standard commercial CAD systems. Our findings and approaches are applicable to 
instructional support for other systems that foster new ways of organizing and thinking about complex 
subject domains. 

Research Background 
The Effects of Previous Experience 

In a previous study, (Shaw, 1992) it was found that architects and engineers were able to learn the 
commands of a well-known CAD system from an embedded tutorial. The program took advantage of the 
system's text and graphics as well as a programming language available through the CAD system to deliver 
instruction, provide exercises, measure success, and display feedback. The scores of the test subjects on a 
drawing test were comparable with classroom taught designers and they felt comfortable with their mode of 
learning. There was a wide range in the time required by the test subjects to complete the lessons and it did 
not correHate with their test scores. The interesting variable in the study was previous experience with 
another CAD system which did correlate with test scores. Those previously exposed to another CAD 
system also completed the test in less time. 

In follow-up interviews (Shaw, & Golish, 1989) the hypothesis that conceptual-level 
understandings occurred in cases that had previous, different CAD experiencewas confirmed. That was not 
true with those who had experience with only the same system or none. These deeper understandings 
included ideas such as the manipulating of complex entities and the algorithmic nature of mirror copying 
objects. Multiple approaches in more than one CAD system gave the learners a varied view of the 
functioning of CAD. This finding concurs with the theory that acquiring advanced knowledge is dependent 
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upon "having a diversified repertoire of ways of thinking about a conceptual topic" (Spiro, Coulson, 
Feltovich, & Anderson, 1988). 

User vs. Machine Controlled Help 

In an effort to artificially provide a conceptual framework for users* an adaptive system for 
administering procedural or conceptual help depending upon experience and behavior in the system was 
tested. The study was inconclusive, but suggested that users preferred to control the type of help they 
received (Shaw,Golish, Webster, & Yang, 1991). The test subjects reported that the word "help" had an 
association with procedural directions and was not consulted if they wanted the other types of assistance. 

As a result of that study, we designed a system that allowed users to select supplementary 
information according to recognized types (graphic, example, practice exercises, procedures, etc.) as they 
proceed through instructional material. We theorized that novices might select a richer and more sequential 
path while those who had more sophistication with the subject would be more likely to design their own 
course and select assistance according to their needs and learning preferences. Preliminary results from a 
pilot study indicate that learners follow the patterns we predicted. 

Tutorial Observations 

To study assistance that might be useful to students using computer-aided instruction for CAD, 
advanced architecture students were observed as they learned CAD from an on-line embedded tutorial (Lee, 
1993). The type of observation was participatory to the extern, that advice was offered when students either 
requested it or behaved as if they needed it. The students were expected to complete a course of study and 
were tested on their skills. The observations included audio and video recording and direct questioning. 
One finding was that students had vocabulary and procedural problems that suggested a richer instructional 
environment might have been helpful for novice learners. The brief tutorial left many questions 
unanswered. 

The study also found that students often could not understand what they had done when they 
executed commands. In the CAD system, many actions could not be directly observed and the system 
messages, if there were any, could not be interpreted by the students. Examples were cited, such as: 
intersections might not join but the error was not obvious on the computer screen; elements might be 
inserted into a drawing but they were not visible in the current screen views; or the computer would report 
that a complex element status had been "exploded" but that was not noticed or understood. 

Meaningful feedback about what the students had accomplished was helpful for novices. This was 
true about productive actions as well as those that might be considered errors. If a student executed the 
correct action, but did not realize it, he or she often repeated the command unnecessarily, sometimes 
placing duplicate objects in the file. Simple feedback information seemed to constitute a valuable level of 
assistance and it might avoid user resistance to active coaching. 



As our effort to provide a rich instructional support system progressed, we realized the extent of 
the task. We had designed over 700 source cards in our basic CAD support system and it was still in its 
infancy. If we were to expand the approaches to include people who use CAD systems for different special 
applications, examples of which range from clothing design to solar mapping, the amount of information 
needed to support users could become staggering. With networking and wide collaboration, a plan for 
incorporating new data could be developed, but the project could never be completed because of the 
continually growing information base. 

As the size of the database expands to become more inclusive, it will become increasingly 
unmanageable for the user. Solutions such as tree-structured menus, "you-are-here" mapping, tracking user 
paths, and "hot" buttons may help, but people who need to learn particular skills or concepts in a reasonable 
amount of time will need assistance just to find their way through the available resources. It is easy to 
become lost just locating a file through the menus in Windows on a fairly large hard drive, and the CAD 
database potentially includes global references from distributed sources. 

It has been suggested that user attributes could be filed and used to set priorities for particular 
types of users. Users could "fill in the blanks" about their characteristics and intent. We feel that would 
require extensive data input, would suffer from "self report" errors, and be difficult to change. Because of 
these problems, users might be excluded from access to potentially valuable resources. Such a system 
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would suffer from inflexibility which is a problem in a dynamic environment. A more valuable type of 
support could be provided by monitoring ucsrs as they work, characterizing and classifying their behavior 
according to recognized user patterns, and using the patterns to determine assistance needs. Machine 
learning algorithms would be able to change as new patterns emerge and would learn about users from their 
work on the computer, in the context of actual documents, not from artificial cases. 

It is likely that the user may not even recognize the need for assistance, but the computer can 
diagnose a problem by two types of comparisons, behaviors with known user models and design products 
with recognized standards. The goal is to offer non-intrusive assistance that would not interrupt the flow of 
ideas (Penzais, 1989) but would contribute information at the right time and in an appropriate way. 
Several difficult problems exist in the process of monitoring, characterizing, user modeling, and assisting. 
We have looked at behaviors that indicate skill level and have attempted .o put them into a cognitive 
framework. 

Indicators of Skill Level 

A machine learning system was based on observations of users as they executed a drawing task in 
a CAD system. The drawing was a plan of a dimensioned staircase. The drawings of users who were 
experts in the same system, other CAD systems, and who were complete CAD novices were studied to 
model differences that could be recognized by a neural network. It was found that there were clear 
differences between the groups in the use of setup commands, translation commands, modify commands, 
and creation commands. For instance, novices used create-erase pairs frequently but CAD experts either 
used or attempted to find commands that would modify an existing element. Those who had used a 
different CAD system required the longest time for the drawing but their standards were much higher than 
the novices. Also, the quality of their drawings was much higher than novices. Our previous study 
(Shaw,& Golish, 1988) would suggest that with greater exposure to the system, their performance might - 
surpass experts in the same CAD software. 

User models were constructed on the basis of their test differences and accompanying verbal 
protocols. Based on the models, a neural network was implemented that would monitor the user behavior, 
recognize the skill level of the user and offer advice to increase productivity. The system mapped from the 
behavior of users to the user model and offered suggestions in a "non-disruptive" manner, flagging the user 
but not requiring attention. The pilot study demonstrated that monitoring user behavior was possible in a 
real world drawing environment (Bhavnani, Garrett, & Shaw, 1993). 

Cognitive Considerations 

Much of the research that we have discussed relates to current work in cognitive science. The real 
world requirements for this work make it subject to much current research concerning situated learning 
(Brown, Collins, & Duguid, 1989). Our studies of on-line help were aimed at delivering real world 
assistance in ways that would effectively fit different tasks, previous knowledge, and preferences. The 
need for understanding the factors that lead to the ability to acquire knowledge in complex domains (Spiro, 
Feltovich, Jacobson, & Coulson, 1992) is essential to this research. Such considerations have inspired our 
observation studies. The constructivist views of the relationship of experience to understanding relates 
directly to adaptive support for learning automated systems. Seeing the world of each person as a 
construction necessitates not only adaptation, but understanding the user's response to adaptation, and 
incorporation of the understanding into the system's knowledge base. Machine learning algorithms, by 
their inductive nature, are appropriate approaches to the dynamic requirements of learning complex 
technology in real world situations. 

Ongoing Research 

The goal of providing performance support in a real world context rather than a "sanitized*' and 
controlled educational setting brought up questions about the complexity of the environment that must be 
understood by the system. What different kinds of domain knowledge and what levels of abstraction must 
the system address? To examine the issues, a group of researchers from the Army Corps of Engineers, the 
University of Illinois, and Carnegie Mellon University met during the summer of 1993. Disciplines 
represented in the group included Architecture, Civil Engineering, Computer Science, Information Science, 
Cognitive Science, and Educational Technology. 

152 



er|c 1 



Smith, D.S. (1994). Machine learning approaches to complex automated systems. In M. Orey (Ed.), Proceedings of the Thirty fifth ADOS 
Conference (pp. 149-156). Nashville, Tennessee: Omni Press. 

Some of the research questions had been explored before the meetings, but at this point it was 
thought necessary to communicate goals and formulate a research agenda for adaptive assistance within the 
concurrent engineering environment. Concurrent engineering, also called collaborative design, is thought 
to be one of the most complex processes humans undertake. It involves many participants through many 
phases over long periods of time and it has a history of poor communication and coordination (Garrett, 
1993). 

As we have stated before, teaching designers to produce high quality documents has been our 
objective for several years. These documents, legally required, arc a convenient starting point for 
automating communication between the different disciplines and time periods involved in concurrent 
engineering. Beyond the questions of CAD performance lie the questions of freeing designers from routine 
problems in operating their systems so they can use CAD as a tool to enhance productive thought. Through 
the process of skillful refinement and modification in collaboration with other professionals, higher-order 
thinking is facilitated. This same principle can be applied in medical, scientific, and other decision making 
environments. 

The results of our think-tank led to research areas concerned with user modeling, strategies for 
automatically acquiring this model, adaptive interface considerations which include both design and 
content, and mapping between the adaptive interface and the user model. The concern for automating the 
acquisition of the user model emerged as primary and seems lo necessitate a machine learning approach. 
The research direction has been multi-faceted, beginning from several directions at the same time. The 
facets include standards of drafting and design (product model), modeling tasks as they exist in the design 
environment (user model), machine learning algorithms (system control and model acquisition), and 
software design tools. The objective is a collaborative effort to develop a rich learning environment for 
CAD that can lead into ;he wider domain of automated design. 

We recognized that much of the work to be done was in uncharted territory and began to search for 
research methodology that would help us ask questions that would provide useful information. Two 
research projects were begun that would lead to knowledge about users' work environments and their 
drawing documents. 

Ethnographic Observation 

To design adaptive support for CAD users, we must understand their work habits and needs. 
Although some studies show the poor performance of CAD systems (Bietz, Langncr, Luczak, Mullcr, & 
Springer, 1990), there has been little work done to study the users of CAD systems in their natural setting. 
Given the lack of information about these users, we turn to the ethnology branch of anthropology to guide 
us in studying their work habits and understanding their needs. Ethnology provides a methodology for 
studying the culture of any specific group. There have been cognitive experiments about capacities and 
time performance on computers (Card, Moran, <& Newell, 1983), but cultural questions, (what is different 
and what is common in the way people interact with a computer application) and their social structure (who 
accesses what information and when) provides a mechanism by which we can test our assumptions in real 
world settings as we design adaptive assistance (Forsythe, 1993). 

The relativist approach to ethnology holds that there is no single description of an event that is 
true, but there can be multiple descriptions of the same event, all being relevant. This along with such basic 
ethnographic tenants as that culture is both learned and dynamic relates directly to our research questions. 
Ethnographic observation also has the benefit of providing us with a clearer picture of the domain 
knowledge of CAD users and may indicate a mechanism for automatically acquiring and classifying data 
about user behavior. 

Following the advice of Carroll and Rosson (1987) our observer visited an office with practicing 
architects using CAD systems in the role of a moderate participator. The first stage was to form a general 
description of the social situation (actors, activities, and the place that they occur). Also, objects, events, 
sequences, goals, and feelings were recorded from interviews and observations. The goal of the descriptive 
stage was to suggest cultural domains. These could be then formalized by semantic classification such as 
inclusion, spacial, cause-effect, rational, location, function, means-end, sequence, attribution, etc. 

According to ethnographic methodology, each stage in the observation process establishes the 
foundation for the next stage, allowing for flexibility and changes in the process. The domain analysis 
isolated specific areas upon which to focus subsequent observations. These were converted to structural 
questions that could then be asked to the informants. On the basis of the information from the focused 
observation, a taxonomy was then constructed according to semantic relationships. After identifying some 
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initial taxonomies, detailed interviews and recording of activities were conducted, followed by 
componential analysis to arrive at a matrix for each domain. 

The results of this observation session are still under study, but preliminary results indicate that 
issues of motivation to leam, maintenance of credibility of a system that might be inaccurate or wrong, 
timing of advice, and the privacy of the user model must be considered. They must be considered in the 
design of a system that not only aims to provide active support for the development of CAD skills but will 
lead to active decision support, symbol retrieval, design case retrieval, standards support, design space 
navigation, and support for collaborative design in automated environments (Bhavnani, Garrett, Flemminu, 
& Shaw, 1994, in press). 

Object Recognition 

One of the recurring questions in offering assistance to designers is knowing the intent of the 
drawing. Just what is being drawn. If it is a road, one type of standard is desirable which is much different 
from the requirement in the layout of a kitchen. A drawing of a building in its concept design stage has a 
much different level of abstraction and, therefore, needs different functionalities of the CAD system from 
the construction drawing stage (use of primitive elements vs. standard symbols or dimensions). All widely 
used CAD systems have stored symbols that have several justifications for their use. It saves time if 
frequently used objects do not need to be re-drawn each time, using the same symbol aids communication 
about the object between users in different disciplines, and many limes there is extensive non-graphic 
information attached to the stored symbols. The trend in CAD is toward increasing use of such symbols or 
"object-orientedness." 

As the number of stored objects in the system increases, it becomes harder and harder for users to 
locate the appropriate symbol. It is often easier for designers to recreate the desired symbol rather than find 
the standard object and insert it into the drawing. Since the use of standard symbols is very desirable for * 
the reasons mentioned above and their use helps recognize the user's intent as he draws, an object 
recognition neural network has been developed that could be used either in a search mode (the user begins 
to sketch the object in a sketch pad) or in a machine initiated mode that "watches" the drawing file and 
offers an object selection when the drawing appromimates a known pattern. The object recognition 
program, AUGURS, has two modules: The first module employs both temporal and spatial information to 
determine where the user is currently working and captures the image of that area. The second module, a 
neural network encoded with geometric knowledge, then performs the classification task on the captured 
image and proposes to the user symbols resembling the captured image (Shaw, Yang, & Lee, 1993, Shaw 
Yang, Garrett, & Bhavnani, 1993, and Yang, & Shaw, 1992). 

While recognition of symbols has application for insuring that drawing files meet standards, it has 
the potential for contributing to user modeling through classifying user intent and stage in design. Changes 
from one type of object to another (wall layout to plumbing) can be noted and classified according to scales 
of abstraction and specific detail. The experiments in this project involve optimizing the program 
performance to extend to a large number of symbols in real time. 

Future Plans 

Standards Checking 

It is difficult to separate higher-order cognitive processes from high quality performance. Perhaps 
this can be explained by the evidence that experts exhibit detailed and extensive knowledge that allows 
them to be creative as well as good performers in their domains (Perkins, & Salomon, 1989). The result is 
that if we foster the development of domain expertise in the use of automated systems we will reap the 
benefits of both high level solutions and productivity. If we discover that the product of a CAD user is 
faulty, we can suspect that the processes that are being used are also faulty. This suggests that if we can 
diagnose the misconceptions or faults as they occur, we may be able to prescribe a remedy and thus 
improve both the process and the product. Thus, an important signal that assistance is needed is marking 
the occurrence of a commonly made drawing error or violation of standards at the time it occurs. 

We plan to test a standards checking program that will monitor a design file in progress, looking 
for commonly observed errors. The program will characterize each element creation or insertion in the 
design file and will simultaneously attempt to offer standard symbol selection through AUGURS. The pilot 
program will offer non-intrusive informational feedback to the user as suggested by previous research fl ee 
1993). * * 
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In future development of the program, behavior monitoring and mapping to appropriate levels of 
assistance will be introduced. Questions about the effectiveness of various types of feedback will be asked. 
Responses other than non-intrusive feedback may be indicated by different user behavior. Such adaptation 
might include instruction or questioning. Also, we hope to identify other user behaviors that could 
contribute to our model. From the combination of drawing verification and behavior monitoring, our study 
of CAD knowledge and user modeling can be made more robust. 

User Modeling 

The user model represents the system's knowledge about its user which is essential to appropriate 
adaptation. The types of information that our model will need to acquire include the user's goals and 
current tasks as well as assessments of skill-level and conceptual understanding. The most important 
machine learning contribution to the user model is the capacity to "utilize ongoing system-user interaction 
to update and improve its user model dynamically" (Rendell, 1994). * 

It is the ability to acquire information automatically that makes a modeling system design feasible 
in the real world. Collecting information for automated systems is problematic. It is time consuming both 
to accumulate data and enter it, but most important, its accuracy is always suspect. Even if the logical and 
user supplied information is correct today, tomorrow's events may make it false. Many times users are no'; 
able to respond properly to questioning and their answers may be incorrect or inconsistent causing incorrect 
behavior of the system. And certainly, users do not like the interruption of queries. Using implicit 
information collected by machine "observation" of the user's actions allows the user to work without 
interruption and permits the data to change over time. 

While a sufficient background knowledge base is needed to begin the inferencing process, 
theoretically, incorrect or outdated information can be phased out and more realistic knowledge can assume 
importance. If a user model presumed that most new CAD users were also new to computers (a fact a few 
years ago) we would continue to waste time teaching the essentials of DOS. The same will soon be true ot 
Windows and then, perhaps, of keyboards. As we have emphasized before, the requirement for 
encompassing change is the essence of a successful adaptive system. 

Our modeling efforts have been mentioned throughout this paper. In this research, attention will 
be given to the interaction of the user model, the domain expert, the assistant or advice component, and the 
system administrator that manages the human-machine environment. Machine learning approaches (Woolf, 
& Murray, 1993) will be utilized to enhance user models, such as determining in which situations specific 
corrective strategies work most effectively. 

Conclusion 

The use of technology will extend and enhance human capacities, but only if people can learn how 
to use extremely complex systems. CAD is a clear example of the type of complexity and layering of 
interrelated domains that can be served by automated tools. Traditional training programs that include CAI 
and classroom treatment are insufficient to cover initial instruction as well as the continuous upgrading 
necessary. The requirement that learning be continual makes the real world the appropriate classroom. An 
often unrecognized capability of extremely complex systems is that, through embedded artificial 
intelligence approaches, they can learn to understand their users and assist human beings when and how 
they need it. This assistance does not end with skill in mastering procedures but can extend to higher-order 
thinking and greater creativity. Our research includes ethnographic and learning observations, machine 
learning algorithms that recognize skill level in users and objects in drawings, and programs that learn to 
assist CAD users to comply with document standards. The key process is machine learning and 
modification of user models that will provide the basis for adaptive systems. 
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Abstract: A design for an adaptive computer-aided training system is described, con- 
sisting of an item bank and a domain independent management system. This paper 
mainly focuses on the design and implementation of the session management sys- 
tem, responsible for steering the session by selecting items based upon the student's 
performance. For implementation, a blackboard architecture is used, in which in- 
dependent knowledge sources communicate via shared memory to reach a solution. 
There are knowledge sources to select a subject, to select a difficulty level, to se- 
lect an i'em, and to maintain the student model. Simulations with a virtual student 
showed promising results. In the near future test sessions will be held on the first-year 
Electronics course. 



Introduction 

It is well known that an important part of learning an engineering subject is solving problems that illustrate the 
principles. Honderd & Jongind (1990) stressed the importance of active participation, i.e. that the students try to 
solve the problems themselves. If the number of students per training session becomes too large, the students tend 
to merely write down the solutions the instructor gives on the blackboard. However, especially in the first two 
years of the education for Electrical Engineer, the courses are massive; there are not enough human resources for a 
more personal approach. 

A computer can offer a solution for this dilemma. If a computer program can handle students with "standard" 
problems, the instructors will be available for students with unusual problems. With a computer based system, the 
students can practice whenever they want. Another advantage is that students seem less reluctant in approaching 
a computer than in approaching a human instructor (Pullen & Mercer, 1988). And last but not least, computing 
power allows simulations and complex calculations to be performed. 

On the other hand, practice has shown that successful CAI systems are not easy to build. Too often systems 
become so complicated that they are only applicable to small domains, hard to maintain or very domain dependent. 
This research is focused on implementation of a domain independent adaptive system with a conceptually simple 
structure. 



Aims and requirements 

The main aims of developing the system are twofold: 

• enable the students to perform exercises as often as they wish and at hours they choose 



• free the instructors from routine work. 
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The requirements can be divided into four main parts: 



Maintainability: An important aim is that the teacher or instructor must be able to implement or change the 
problems by himself/herself in an easy way. It should take little effort to keep the system running. Further, since 
there are a lot of problem types and every teacher has his own preferences, there should not be a restriction to a 
single problem implementation tool. 

Domain independency: Because of ihe intended use for various courses, the system should be as domain 
independent as possible. This means that the domain dependent information must be kept separately. 

Usage of graphics and simulations: With engineering courses, graphics are often used to clarify subjects. In 
control theory for example, time- and frequency responses, block diagrams, and pole/zero configurations are used 
intensively. Therefore, the instruction system must be able to display graphics and perform simulations. Since 
good simulation software is available already, usage of existing software is desirable. 

Tailoring to individual students: To be able to handle students in various preparation phases, the system should 
adapt to the level of the student. This means that the system must support various levels of difficulty. The 
student should get problems that are not too hard and not too easy. Also, the problems should support variation 
of parameters to be able to create a different instances from the same problem. Accuracy of the adaptation is less 
important in a training environment than it is with tests; the primary goal in training is to offer practice while the 
primary goal in testing is to give an objective and fair measure of the student's proficiency level. 



In view of the requirement of domain independence, the concept of item banking is used. An item bank is a data 
base containing items and information about the items. Each item represents a specific problem. An item manager 
provides an environment for creation and maintenance of items and item bank management. A session manager 
uses the item bank to create training sessions for the students. 

One of the drawbacks of the currently known item banking systems is that item constructors are limited to the 
environment the system poses onto them. The experimental system Items-D (Jansen et a!., 1991) does not provide 
an environment for item creation itself, but relies on external environments, allowing the item constructors more 
freedom. Unfortunately, Items-D is nonadaptive; all sessions have to be specified beforehand. The system described 
here will use the Items-D method for item implementation and item bank management, but will provide an adaptive 
session manager. A description of the item bank structure and the session manager architecture will be described 
in the next sections. 



The item bank 

The item bank is the domain dependent part of the system. Items are usually problems, including execution 
specifications and possible data files, and information about the problem. This information consists of, for 
example, the subject it is about, the difficulty, the variable parameter information and the methods to calculate 
the answers. In addition, the item bank contains a list of subjects that are involved with their mutual relations 
(prerequisites, curricular order etc.). 

The items must be able to read and create data files containing parameters, answers and other relevant information 
like the maximal number of attempts and whether or not help is available. The interface definition from Items-D is 
adopted, making it possible to use the items also in standard Items-D. This method uses ASCII files for data transport 
between the management system and the item. Items can be implemented in any programming environment as long 
as it is possible to generate and read data files, and the environment allows stand alone execution of the application. 



System architecture 
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The session manager 

In the aims it was stated that the system must adapt its training at the student's course level. The task of the session 
manager can be divided into several subtasks: 



• select subject 

• select difficulty 

• select item 

• show item and get answer(s) 

• update student model 

A control mechanism will have to take care of the initial chokes, stop criteria and the activation of subtasks. These 
fubtasks need shared information like the current subject, difficulty and the est.mat,on of the student level for the 
current subject Further, the order of execution of the tasks is not explicitly known >n advance. It is poss.ble fo 
eSe that the difficulty selector concludes that it is better to change subjects, in which case a new subject (and 
ft?ta rt*d t a new difficulty) must be selected before an item can be chosen. In this s.tuat.on a blackboard 
architecture (Jagannathan et al., 1989) seems a suitable solution. 

The architecture is best described by a metaphore. Imagine a number of independent experts cooperating to solve 
a^ro^m^pS have experL on different fields relevant to the problem. They use a blackboard to vvnte 
down . h Tgoah ^and the intermediate results. A chairman can be assigned the job of keeping structure ,n the sol v.ng 
p£» see figure 1). An expert informs the chairman if the information on the blackboard g.yes poss.b.ht.es tor 
So" of new results. The chairman allows the expert that has the best chance of coming closer to the des.red 
goal to use his expertise. 
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Figure 1 : The blackboard model 



Applied to our problem of giving proper training to the student, the final goal is that the student has a good 
impression about his abilities to pass the exam. The subtasks specified above form the experts. Each expert or 
knowledge source (KS) has an activation record (KSAR) which holds preconditions for activat.on and possible 
postconditions. For example, a KS for selection of an item has as preconditions the availability of subject and 
difficulty, and as postcondition the availability of an item. The analogon of the chairman .s a control procedure; 
the blackboard is a collection of shared data. 

159 



ERIC 



167 




The system adapts on two entities: the current subject and the current difficulty level. This section explains the 
idea behind the adaptation processes in more detail. 

Subject adaptation 

A structure called a subject lattice is used to find a suitable subject. This structure specifies the relations between 
all relevant subjects of a course. A prerequisite link from subject A to subject B shows that the student should 
master subject B sufficiently before before subject A can be selected. Further, a curricular order can be specified 
showing the "ideal" order of the subjects. An example is shown in figure 2. Arrows denote prerequisite relations; 
the numbers show the curricular order. 



Difficulty level adaptation 

The difficulty level adaptation uses item characteristic curves (ICC's), originating from the item response theory 
(IRT). Before the adaptation method is explained, a brief description of the item response theory will be given. 

According to Hambleton (1989), 1 An item response model specifies a relationship between the observable examinee 
item performance (correct or incorrect responses) and the unobservable abilities assumed to underly performance 
on the test. This relationship is called an Item Characteristic Curve". The ICC is described by a mathematical 
function. The value of an ICC for item i at ability level 0, denoted Pi (0), can be viewed as the probability associated 
with a randomly selected examinee at ability level 0, answering i correctly. An often used model is the 3 parameter 
logistic model (3-PL), defined by 



in which the three parameters are the item difficulty 6*, the item discrimination parameter a* and the pseudochance 
level Ci. The influence of these parameters is shown in figure 3. 

For proficiency estimation, the likelihood function L is used. This function describes the likelihood that a student 
with proficiency 0 would give the response pattern a (in which 0 = incorrect, 1 = correct). 




Figure 2: Example of a subject lattice 
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Pruikiency 

(a) 

Varying a (b=0, c-0) 



Proficiency 



(b) 

Varying b (a=l. c-0) 



Proficiency 



(C) 

Varying c (a-l. b=0) 



Figure 3: Examples of Item Characteristic Curves with 3-PL 

In words, each term in the product is an ICC if the corresponding answer was correct and the negation of the ICC 
(1 - ICC) if the answer was incorrect. If the item parameters and the item scores are known, the value of the 
likelihood function only depends on the unknown variable 6. By maximizing the likelihood function with respect 
to 0, the maximum likelihood estimate is found. For more detailed information about the item response theory and 
adaptive testing see Hambleton (1989) or Wainer (1990). 

The IRT method has the advantage of making optimal use of available information, but it is computationally very 
intensive. Further it depends on unidimensionality, i.e. the same trait is assumed to underly all items. With various 
subjects, our application clearly does not satisfy this condition. Extension lo multidimensionality would require 
an even greater computing power. And does the method take into account that the student learns from previous 
items? With tests this learning effect is probably not so high, but with instruction you want the students to improve, 
so the student's level will not remain constant. Another point is that IRT selects a next item that gives the most 
information. But from a pedagogical point of view, this does not always have to be the best item. In instruction, 
one would probably prefer a gradual increase in difficulty above a switch from very easy to very difficult as could 
easily occur in the standard IRT after asking one item: if the first item is simple and is answered correctly, the most 
likely level is +infinity, yielding the most difficult item available. 

The method proposed here intends to select an appropriate difficulty level that is less computationally burdensome 
(but also less accurate). The new difficulty level is calculated as follows: 



b = scale {Pr l [(l-Pc) xi - Pi 1 '*']} 



(3) 



The function scale transforms the argument from the interval (-00,00) to the interval 10,1], P { 1 is the inverse 
function of Pi. The (3 factors describe the speed of adaptation; f3 c is used with correct answers and (3j with incorrect 
answers. If they are close to zero, the adaptation will be abrupt; close to 0.5 there will not be any adaptation at all. 
Empirically found reasonable values are between 0.01 and 0.2, depending on the quality if the items. 

With a correct answer to item i (xi = 1), the method takes the level that gives al-fc chance of success on 
i. Similarly, with an incorrect item, that level is taken for which the probability of correct response is If a 
guessing floor c is used in the ICC, (3j must be greater than the largest c because the minimal probability is c. 

The advantage of this method is that it needs only the current item's characteristic curve and can be computed very 
fast without the need of numerical methods. Another point is that earlier results on the subject are forgotten more 
quickly. This means that if a student "sees the light" on a topic, his past won't hunt him very much. 
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Session manager implementation 

The blackboard 

The information on the blackboard can be divided into several parts: the domain model blackboard, the student 
model blackboard, the steering blackboard and the control blackboard. 

The domain model blackboard contains the previously described subject lattice. The subject contents are not 
specified in the doma'n model blackboard; these remain within the items. The item bank is also not part of the 
blackboard, as only .ue item selector needs access to it. The information of the current item however, if any, is 
available on the blackboard. 

The student model blackboard contains a reflection of the domain model. Per subject it contains an estimated 
proficiency level 0 (with a moving average 0), and a notation for the student's assumed competence, with possible 
values Unknown, Perhaps acquired, Possibly acquired and Strong. This notation was introduced by Lesgold e.a. 
(1989). 

The steering blackboard contains pedagogical information like the criteria the student must meet to qualify for 
having "sufficiently mastered" a subject (for example, the status is at least Probably). Further the speed of the 
adaptation process and the maximal number of items on one subject are available. At the moment this is a write-once 
and then read-only blackboard, but in the future a monitoring KS could be allowed to adjust the parameters. 

The control blackboard contains information about wanted and available (intermediate) results. This is used 
by the knowledge sources to check if both their, preconditions are available and their postconditions are wanted. 
It consists of two flag pools: a "wanted" pool with results that, if obtained, can lead to a new solution, and an 
"available" poo! containing already obtained results. KS'es are free to examine and change these pools, thus 
possibly influencing control decisions. For example, if the subject model updater notices that the proficiency level 
on another subject decreases because of the item result, it can remove "subject" from the available pool. Then, 
before a next item can be selected, a new subject will have to be chosen. The control mechanism is also able to 
update the control blackboard. If no experts are triggered by the available information, it looks for unsatisfied KS 
preconditions and posts these as "wanted". 

The knowledge sources 

The subject KS takes care of subject adaptation. First, it exan rnes the lattice for subjects with state Perhaps. This 
is a nonstable state, showing that more information is needed to be able to decide whether or not the student's 
knowledge is sufficient. If there are no subjects with state Perhaps, focus is put on subjects without prerequisites 
or with sufficient student mastery on the prerequisites. If this still does not give a result, then the reinforcement 
technique is applied: subjects with state Probably are made candidates. Normally this means that the subject lattice 
is traversed again in curricular order. If the selector still was not able to select a subject now, it signals the control 
blackboard that the subject lattice is exhausted. An extra parameter taken into account is the number of items asked 
on a subject. If a student's state on some subject is Unknown after 5 relevant items, there is more reason to doubt 
that student's knowledge than after one item. Naturally, other strategies could also be applied, for example first, 
raising a status to Strong before selecting a new subject. However, this has the risk that the students stay with a 
subject for a long time and get bored. 

The difficulty level KS selects a new difficijlty with the method described in section "Difficulty level adaptation" 
above. Simulations with virtual students of various proficiency levels have shown that this method gives the desired 
adaptation to performance. This also holds if the difficulty level is a discrete variable with a small number of 
possible values (e.g., 10) as will be the case in reality: for teacher convenience the difficulty level will be described 
in fuzzy terms like "hard", "easy", etc. 

The item selector KS searches the item corresponding best to the currently specified subject and difficulty level, 
taking into account restrictions like "don't give the same item twice in one session". Per item (problem) the main 
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subject involved and possible influencing sub-subjects are recorded. Each item subject is labelled with parameters 
characterising an ICC (difficulty level, discrimination value, guessing correction). If an item is selected, the 
characteristics (subject records with ICCs) are put on the blackboard. 

The item presenting KS presents an item to the student. It runs the executable cf the item, after preparing the 
necessary information, Items-D compatible data files, containing parameters, answers and possibly various other 
information like number of tries, additional short hints, rules for grading the item, etc. These are all available in 
the item specifications. The executed item returns a data file containing the result (given answer(s), obtained score, 
number of retries) which is interpreted by the item presenter. The important results are reported on the blackboard. 

The student model KS updates the "shown levels" (the levels at which the student has shown to answer an item 
correctly) and, if needed, the state variables of all involved subjects. A state change from Unknown to Perhaps 
occurs if at least one item of at least a minimal level is answered correctly. A state change to Strong occurs if 
the moving average over 3 items is above a maximal level Then, if the level is above a medium level, the state 
becomes Probably; if the level is under that medium level, it becomes Perhaps. The student model updater also 
signals the control blackboard if a relevant (i.e. subject changing) state change has occurred. 



Item implementation 

A number of items has been realised for introductory courses on Control Theory and on Electronics, using the 
authoring system Taiga (Educational Centre, 1987) with the toolboxes PTznd the Control Toolbox, developed at 
the laboratory (Essenius, 1992). These toolboxes process interface files and offer easy methods to display several 
graph types like time- and frequency responses, pole-zero configurations, flow diagrams and drawings made in an 
ext aal drawing program. 

An item acceptance test in the fall of 1993 showed that most of the students appreciated the computer training. 
However, the item layout in Taiga needs special consideration. Because of the limited display facilities (maximal 
text size is 20 lines with 80 characters 20 lines) the display tends to become obscure. Figure 4a shows an item 
implemented in Taiga. At the moment of the screen capture the solution is shown. 





Figure 4: Examples of implemented items 

With Inigo (Educational Centre, 1992), the successor of Taiga, this problem was solved by using the environment 
MS-Windows. Taking into consideration other advantages of MS-Windows in comparison to MS-DOS, it was 
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decided to "upgrade" the items into Inigo. The interface file processor has been ported to Inigo and currently the 
Control Toolbox is being ported. 

Some items on Control Theory, involving simulations, were made using Turbo Pascal 5.5. These items use the 
transformation and identification package TRIP inline to calculate responses. With this technique, it is possible to 
play "what if" games with the student. For example, figure 4b shows a problem in which the student is asked to 
enter a pole-zero plot corresponding to a given time response. If an incorrect answer is given, the item can show 
the time response that would be obtained with the answer. More information is given in Essenius et al. (1993). 



Future research 

It is considered important that the student has some influence on the decision process. Student enquiries also 
supported this viewpoint. However, learner control often does not give satisfactory results (Chung & Reigeluth, 
1992). Weaker students tend to practice less than they need and better students sometimes practice much more than 
they need, just to be certain. The possibility for computer guidance, in which the system offers a number of choices, 
suggests one and leaves the final decision so the student, could offer an alternative to strict computer control on 
one hand and strict learner control on the other hand. The possibilities to implement this are now investigated. 
At the moment, the final parts of the session manager (the item displayer and some parts of the item selector) are 
being implemented. In the end of March a first test (under MS-DOS, still with Taiga items) will be organised with 
volunteering students. After evaluation of the test and a teacher enquiry, a larger scale test under MS-Windows, 
also with Inigo items) will be performed. The first release of the system is planned in October 1994. 
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Deciding What to Remediate in an Intelligent Tutoring 
System: a Practical Approach 



Abstract: An Intelligent Tutorial System (ITS) provides adaptive remedial 
tuition by making inferences about the nature of the student's misconceptions, 
typically by maintaining a model of the student's knowledge and comparing 
it with an ideal model. Such models are complex. ICATS is an ITS which 
interfaces with almost any Authoring System, to provide an adaptive environ- 
ment based on a Course Hierarchy Network (CHN). The CHN is a directed 
graph representing the dependencies between the subtopics constituting a les- 
son. ICATS also maintains a Student Progress Record (SPR) which tracks 
the students path through the CHN. The nodes of the SPR constitute a sub- 
set of the nodes of the CHN, but contain different information. This paper 
briefly describes ICATS> and then discusses ways in which the SPR may be 
used to diagnose a students weaknesses, and provide remedial tuition. The 
paper demonstrates the possibility of achieving adaptive behaviour in a CAL 
system, without complex knowledge representations. 



Much current research and development in Intelligent Tutoring systems (ITS's) is based on the use of many 
complex representations, such as: the knowledge domain, models of a student's knowledge, representation 
and classification of student errors etc. The "Lisp Tutor" (Reiser et al., 1985), and "PROUST" (Johnson 
W.L et al., 1987; Littman et al., 1988) are examples of this type of system. A number of other examples 
are discussed in (VanLehn, 1988). 

Such efforts are highly desirable, and indeed necessary, if we are to achieve ITSs that approach the 
usefulness and adaptability of human tutors. 

Meanwhile there is much useful work that can be done using cruder techniques that nevertheless make 
Tutorial Systems more intelligent than conventional Computer-Assisted Learning Systems. This paper 
describes such a system: ICATS (Intelligent Computer-Assisted Tutorial System). 



ICATS is a development of an earlier proposed system (Harris, 1985). ICATS uses a directed graph, the 
Course Hierarchy Network (CHN), to represent, crudely, the knowledge of the domain, as well as a crude 
model of the student's knowledge, the Student Progress Record, to determine what remediation should 
be applied, and when. 

ICATS allows the courseware author to define the CHN corresponding to the prerequisite structure of 
the topics making up a lesson. Each topic can then be authored with an appropriate authoring system. 
ICATS links the authored segments to the nodes of the CHN. The author supplies information about 
pass-mark values, and any other supplementary information. More details about the CHN are discussed 
below. 

The use of a graph to define the structure of lesson material has been discussed in (Ferraris et al., 1984) 
where a Petri net (Peterson J.L, 1981) description of the lesson is advocated. However, unlike ICATS) 
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Ferraris's system does not suggest using the net to control automatically the sequence of CAL material, 
or to determine the need for remediation and what remedial material to apply. A further difference is 
that Ferraris's net contains loops, whereas the 1CATS net does not. ICATS assumes that if a given item 
of teaching material depends upon knowledge of earlier material, then no earlier material can depend, in 
turn, on it. This principle rules out loops. 

Moreover, the Petri net must be interpreted as a dynamic representation of the CAL material - 
hence repetition of earlier material must be represented by a loop. The ICATS net represents the static 
dependencies of the material, and the ICATS inferencing and control mechanisms activate forward and 
backward movement through the net. 



The purpose of the CHN (Fig.l) is to represent the hierarchical, or prerequisite structure of the CAL 
course. Each node (Fig.2) in the CHN represents a segment (see below) of CAL material. 

The segment 

A segment would, typically, be a screenful of orthodox CAL material such as would be produced by 
any decent Authoring System. The node might include an executable CAL filename, or a pointer to 
executable CAL code, or a procedure that invokes a CAL segment: the implementation details are not 
important. For the purposes of this discussion assume that a node is an abstract entity that, somehow, 
under a set of conditions to be explained below, triggers a CAL segment. 



Each node, except the start node, has a non-empty set of immediate "parents". For any given node 
its parent nodes represent the CAL segments whose knowledge content is assumed by the given node's 
CAL segment. Parents of a given node can be associated logically by AND or OR relationships. 1 



Number in node: identifier i; italicised number: depth g of node. S: starting node; T: terminal node 
Figure 1. A simplified, but typically structured, CHN 

If each CAL segment is "fine-grained" - i.e is confined to imparting a very small piece of knowledge 
the CHN would correspond, loosely, with a representation of the knowledge domain. However, since 

*NOT is unlikely to be relevant for CAL Applications, but perhaps should be included for completeness. 



The Course Hierarchy Network (CHN) 



The node 
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the CHN is a directed graph without loops," it would not be exactly equivalent to a representation where 
knowledge-elements were linked associatively or semantically. In fact the CHN represents meia-knowkdge 
of the knowledge domain. Strictly speaking, the CHN represents the course designer's knowledge of the 
dependencies between the segments. 

In addition to the segment information, each node in the CHN is associated with information that 
is used to determine the attainment-level of a student when s/he interacts with the associated segment. 
Again, the actual Implementation details are of no concern: this additional information may take the 
form of node-local and/or system-global rules applied to the actual student input, or to data supplied by 
the segment, such as the scoring methods to be used for the segment. 

Other segment-related information may also be attached to the node, such as: relevant remedial 
material, reference material, hints, and so on. In the interests of separating abstract representations from 
CAL-specific data, the associated information should take the form of pointers to, or indirect invocations 
of, rather than direct calls to such material. 



i 8 
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CAL 
Segment 
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Synopsis 



The number in i is the node identifier 
The number in g is the depth 
The numbers in the other boxes are pointers to parent or child nodes 
Typical node structure 
Figure 2 

A Practical Example 

The WATS CHN 

The following describes each node in the CHN, in the ICATS application. 
The JCA TS Node 

Any node M is a 6-tuple, such that; 

M={i,9,V,V,S,A} 

where: 

V is the set of immediate parents of fif\ 

V is the set of child nodes of tf\ 
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S is a boolean expression, representing the prerequisite relationships for jV, such that each element of V 
appears at least once in £; 2 
t uniquely identifier this node; 

q is the node-depth; i.e the number of nodes in the longest direct path from the start node to Af; 3 

A represents all information associated with N; 

for ICATS, ,4 is a triple: 

A={s,m,t} 

where 

s identifies the associated CAL segment; 

m represents a pass-mark for the associated segment; 

t represents a "threshold" for the associated segment; 

let v be the mark obtained by the student when the segment was attempted: 
then 

v<m represents failure; 

m<v< t represents a "weak" pass; 

v>t represents a "good" pass. 

Implementation Considerations 

In ICATS, the node items are held as pointers to data or to lists, or to procedures, so that "raw" 
data is kept separate and the CHN remains an abstract representation at a very general level. 

The ICATS Student Progress Record (SPR) 

Each student has an SPR. The SPR is a set of data-records, ordered in accordance with the order 
in which the student tackled the CAL segnv 'ts. 

The SPR Data-record 

Each data-record d in the SPR is a tuple: 
*={s,v} 

where s and v have the meanings described in The ICATS Node, above. 

Strategies for Remediation 

There are several distinct strategies for identifying the origins of failure and thereby determining what 
remediation should be adopted. Each strategy uses the information in the CHN and SPR. Generally, 
each strategy assumes that the cause of failure (in a segment associated with a node) can be attributed 
to weakness in one or more nodes along one or more of the paths that led to the failed node. However, 
each strategy uses the information differently, these strategies are discussed below. 

The Maximum Depth Principle 

ICATS operates on the Maximum Depth Principle. That is, the system always delivers, when possi- 
ble, the deepest CAL segment that the student is qualified to attempt. Depth is measured by g (see 
above). 

This principle and the strategies for remediation (see below) are based on the intuitive beliefs and 
experience of the author, rather than on any established pedagogical principles. 

Justification of the Maximum Depth Principle 

It is in the interests of the student to keep her/him progressing as deeply into the material as possible. This 
helps to maintain interest, "stretch" the student, and to develop a deeper understanding of the subject, 

2 The presence of € makes the need for V redundant. It is retained for "housekeeping" purposes. 
3 <? is used with the Maximum Depth Principle; sec below 
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that if remedial back-tracking is required, s/he can re-attempt earlier material with greater confidence. 



so 



Definitions and Notation 

J/ is any node in the Student Record which is not a weak node (see below) and is not a failed node. 
T is any failed node in the Student Record. 

X^T represents any path from hi to ? t consisting of just those nodes in the Student Record which 
were traversed as the student progressed from hi to 7, inclusive of hi and T. 

There may be several distinct paths hl->F; that is, several paths hl->T containing nodes associated 
with segments "passed" by the student, which converge on T. 

General weak path. (GWP) 

A path is a GWP iff it is either a CWP (see below) or an AWP (see below). 



© 0 © 




G: "good" pass; W: "weak" pass; F: "fail"; dashed line: weak path. 

Figure 3 



ERIC 



Contiguous weak path, (CWP) 

is CWP (Fig. 3) iff all nodes in Af->? other than M and T are weak nodes. 
Absolute weak path. (AWP) 

A path is AWP iff it conforms in every respect to a CWP, except that there is no node in the Student 
Progress Record that corresponds to N. 

In other words, an AWP is a path from the start node S , to T % in which all nodes (including S) up 
to but excluding the immediate parent of T are weak nodes. 

Weak node 

Any node with associated score v in the SPIt, where m<v<t (see above), is a weak node. 
Earliest weak ancestor. (EWA) 

For any GWP, the weak node nearest to S (or S itself, in the case of AWP) is called the EWA 
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ofjF. 

Latest weak ancestor. (LWA) 

For any GWP , the immediate parent of T is called the LWA. 
Mutual weak ancestor. (MWA) 

For all GWP's in the Student Record, any node which is common to two or more such GWP's is 
called an MWA, 

Convergent Mutual Weak Ancestor. (CMWA) 

For all GWP's which converge on a given T> any node which is common to two or more such GWP's 
is called a CMWA. 

Degree of mutuality 

The degree cf mutuality, of any MWA or CMWA is measured by the number of GWP's that share 
the MWA or CMWA. 

Possible Remediation Strategies 

EWA Strategy 

1. Find the earliest failed node. 

2. Find any GWP for the failed node, and its associated EWA. 

3. Re-issue the material (original, or, if present, remedial) associated with the EWA. 
EWA : Justification 

The GWP probably led to the failure in question (the earliest failed node). 

If the EWA can be successfully tackled by the student, there is some hope that each succeeding 
weak node in the GWP can then be tackled successfully, since each weak node is dependent on its weak 
predecessor. 

Hence the earliest failed node should be taken first, since subsequent work by the student will often 
lead to her/his being able to grasp earlier, usually more elementary, points, with greater confidence. 

LWA Strategy 

Essentially the same as the EWA Strategy, except that the LWA is sought. 
LWA : Justification 

Subsequent work by the student probably boosts her/his confidence and general grasp of the subject, so 
that s/he will probably succeed with the LWA without having to work through the entire GWP. 

However, if the student fails the LWA again, this strategy will loop infinitely round the LWA, unless 
the fact of second-try failure is noted (see Failure Count, below) and used so that next time the weak 
parent of the LWA is tried, and so on back up the GWP. This is a more complex strategy than the 
EWA Strategy. 

MWA Strategy 

All MWA's derivable from the Student Record are noted, with their degrees of mutuality (see above), 
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The MWA with the highest degree of mutuality is issued. 
MWA : Justification 

If a weak node is common to more than one failure, it seems likely that it is responsible for all those 
failures, or, at least, has contributed substantially to them. 

Moreover, the one with the highest degree mutuality is probably the most culpable. 

CMWA Strategy 

As for MWA , but with CMWA's, 

CMWA : Justification 

The justification for this strategy is similar to, but stronger than that given for MWA. 
Combination Strategy 

Combine the above strategies. For example, if the MWA Strategy still leads to failure, try the LWA 
Strategy, and if this is unsuccessful, try the EWA Strategy. 

This will require maintenance of a record, within the SPR, of what strategies have been used with 
each failed node. 

Alternatively, the strategy could be tied to a Failure Count (see below), so that, as the value of the 
count changes, another strategy could be tried. 

Failure Count 

Each failed node in the Student Record could carry an associated failure count, indicating how many 
times the given node was failed. 

The system could limit the number of failures allowed (perhaps as a teacher-supplied option). When 
the number is exceeded, it would indicate to the student what the areas of weakness were (e.g: show 
the synopses associated with each node in the given GWP), and what reading is needed to remedy the 
weakness. 

Conclusion 

This paper has attempted to demonstrate that strategies exist that would permit the use of conven- 
tional CAL material within an ITS, such that reasonable inferences may be made which identify where 
remediation should occur, without a complex knowledge representation. 
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Abstract: Among the mandatory courses of the undergraduate curriculum in 
Computer Science at Universite Laval, there is Systems analysis and design. 
Conceptual database modeling, and more specifically Entity-Relationship approach 
(Chen 1976), is an important topic covered in that course. The production of an 
Entity-Relationship Diagram (ERD) that actually represents the access paths to data 
needed by an enterprise, is a very complex transfer task. Leshin, Pollock, and 
Reieeluth (1992) state that transfer tasks have great variation from one performance 
to another. These tasks cannot easily be broken down into steps, because each time 
the task is performed, the learner generates a procedure which is appropriate to that 
particular set of circumstances. 



Introduction 



Systems analysis and design is amandatory ccirse of the undergraduate ^curriculum in Computer Science 
atUniveVsite Laval. An important topic is modeling data with En^f^ 0 ^^^^^/^. of ^ 
Data modeling is not easy. The learner's task in data modeling is to figure out what types of dato 
underlie the business function under study. The learner must also capture a replica of dial structure. Eacn 
Scss fuLtonTdiffcrenL Leshin, Pollock, and Reigeluth (1992) ^stated 1 that: "Transfer ^ks ^have jreat 
variation from one performance to another" (p.49); "... cannot easdy be broken down jm»o stcps^ because the 
activUv varies each time the task is performed" (p.82). Elaborating an ERD corresponds to a transler task 

y SS^SwA« course has two parts. Concepts, principles, and rules are taught during a 
meoretSSTeamS uansfer this knowledge during a practical part; they have to resolve cases. The 
COnSpmal rStabSSehng Advisor (CODAMA) was developed to further the transfer of knowledge during 
the practical part 



Advisor system 



An intelligent advisor system is an adaptive system aimed to intervene when an individual is pe^j™ng 
a complex transfer task (Boulet, 1992). This instructional technology takes place after the unUal learning of 
^te?pSW«^ l rules. It is an environment of help utterly linked to the use of what ^ ^nleamed. 
Users think and integrate several kinds of knowledge to find the "right" way to solve a comp ex problem. 
Users unnk ana ^ ^ ^ ^ ^ i(tentifl cation of strategies and tactics the advisor uses 

to provide adviS guidance, hints, explanations, and to answer questions. The main purpose «s to provide an 
rnvionment tailored to the user's needs and goals. An advisor system helps either to relate theory to practice, or 
to develop more effective problem-solving strategies, or both. 



CODAMA 



Students enrolled in Systems design and analysis course have weak knowledge of the domain and no 
practical exSnce. 'I ney draw their ERD with a CASE tool (Computer Aided Software Engineering, Figure I). 
CODMlAKteSaced to that CASE tool. The set UP of CODAMA adds an 2^^"^ 
ADVISOR option. A learner produced a part of an ERD She or be ^^^^ 0 ^^^T^ 
ADDRESS, STUDENT, and PROFESSOR, and the relationships ENROLLS, GIVES, ,^ ^1^ arc 
attributes not yet associated. Some cardinalities was set The learner can call CODAMA to make sure that this 
part of the ERD actually represents the business function. 
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Figure 1. CASE tool window. 

Figure 2 shows the menu displayed after a learner told CODAMA that she or he warned explanations 
about cardinalities of the relationship ENROLLS. 



FOe Advisor 

r 



STUDENT 



Passive mode: CARDINALITIES 



□ Explain 

□ Expiain 
□Explain 

" O Explain 
□Explain 
□Explain 

□ Explain 
□Quit 



couple 0 t n-0,n 
pair 0,n of entity STUDENT I 
pair 0,n of entity COURSE 
minimum cardinality 0 of entity STUDENT [ 
maximum cardinality n of entity STUDENT | 
minimum cardinality 0 of entity COURSE 
maximum cardinality n of entity COURSE 



■[ggggn 




#ldent. prog, 
f'rofcram name 


fluent prol. H 
Name [J 







Figure 2. Example of menu. 

Suppose the learner selects Explain minimum cardinality 0 of the entity STUDENT. Hie explanation 
presented at Figure 3 will be superimposed. There are underlined words. Two of them refer to prerequisites- 
f*'Z Relationship. That means an individual cannot understand what is the meaning of this explanation if 
SSS- h £T 01 *** n0t rcmcmbcr what is enUty or a relationship. By clicking on one of those 
underlmcd words, the learner can have the corccsponding explanation (or a set of expiations because f there Tc 
SUi nderhned words within other explanations). This access to prerequisites was set to help recalling one or 
another concept in the short term memory. s 
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*IIC1UT11CSM)I 



Minimum cardinality 0 of entity Course 
means that a Course does exist even if it U 
not given during a trimester. It must be a 
T ist nf fnurscg available within this 
organization. Is it true that entity Course 
does exist without having to participate to 
the relationship Enrolls? 



(YES) <$o^ 



Figure 3. CODAMA's answer. 
The learner can also see and use a network. A part of the network is presented at Figure 4. 



Maximum 
cardinality n 




Relation name 



Relation^ 



Entity 5T^ Functional 

identifier *-| mtegnty 

1 i J I Eomy JfeH Properties constraints 

T^tx T 1 — "- — 1 

Minimum 
I cardinality 0| ] Entity name 



i r — •» 

■ 

Figure 4. Links between prerequisites. 

A network can be used in many ways. A learner can ask for the network showing prerequisites for a 
given concept. She or he can use it to have a given explanation by clicking on one box. Another learner can ask 
which box corresponds to a certain question she or he asked CODAMA. She or he can also see which box 
corresponds to a particular mistake that the advisor detected. Doing so, she or he will also see what are the 
related prerequisites. 

The third underlined group of words, i.e., List of Courses (Figure 3), refers to a hypothesis that 
CODAMA makes. If the student clicks on those words, CODAMA will present a list that is supposed to exist 
within the organization. Note that, for some other cases, CODAMA will present hypothetical forms such as a 
bill. This allows the student to relate her or his perception of the reality to the entity-relationship formalism. 
That is what an analyst MUST do during all her or his professional life, i.e., verify repeatedly whether what he 
or she models is the real thing. 

An advance organizer (Ausubel, 1960) can be used before starting the process of elaborating an ERD. 
First, CODAMA displays a complete and real ERD obtained from experts. A knowledge acquisition module 
allows the collecting of those real ERDs. It is described in the next section of this paper. The ERD being 
displayed, the organizer simulates the elaboration of this ERD. The simulation is made in accordance with a 
hierarchy of prerequisites 

When the advance organizer module intervenes after a user begins to elaborate an ERD, it takes into 
account the learner's cognitive structure. There Is a continuous collection of data that is invisible to the learner 
questions asked and errors made ate categorized and recorded 

Suppose that a learner used several menus but did not use keywords. CODAMA will fit its 
interventions. It could produce the advice presented at Figure 5. Following this advice, CODAMA begins a 
simulation of data modeling. First, it refers to the highest element of the hierarchy consulted by the learner. 
Then, CODAMA goes down the hierarchy to set the content of its next explanations. In other words, it presents 
the prerequisites. When the bottom of the hierarchy is reached, CODAMA goes back all the hierarchy. In the 
particular case illustrated at Figure 5, CODAMA's simulation puts more emphasis on explanations related to 
fundamental concepts and principles and less on the modeling technique. 



175 



1S3 



Boulet, M.M. (1994). Adding advice to a CASE tool. In M. Orey (Ed.). Proceedings of the Thirty-fifth ADC1S Conference (pp. 173-177) 
Nashville, Tennessee: Omni Press. 



There are 60 property ia the ERD; you 
I asked 1 1 times how to find properties. 
| There are 11 entities you asked 4 times 
bow to And entities. There are 16 
relationships: you asked 6 times bow to 
fed them. You never review concepts and 
principles. In my opinion, you do not 
master those fundamental concepts and 
Principles. 



Figure 5 Example of advice. 

The knowledge acquisition module 

The knowledge acquisition module allows the recording of real ERDs. To acquire those examnles 

hkjj is then explained. An example is presented at Figure 6. 



'Collecting comments about analysis 
Analysis of: 0 t n - l,n -> Case 1 
CUSTOMER 0 t n OWNS l,n HOUSE 
Kind of wganization: INSURANCE COMPANY 
Which information does the INSURANCE COMPANY 
need to memorize in regard of this couple of pairs of 
cardinalities? 



| EXAMPLE AVAILABLE 




Why does the organization need this information? 


1 1 




roKj 


(Example) 


Cancel) 



Figure 6 Example of expert's comments 

In tie following example, what an expert wrote to explain a minimum cardinality 1 is placed between 
^JST ° < T ul ^ e S u€ 0i >"»'«">• Its clients are Jetail shops, persons, and organfzaTns? ms 
enterprise <makes to order>. When it receives an <order>, the following form is used (Hgure 7)T 
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Figure 7. Order form 



pmiiies <Order> and <Furniture> and relationship <Concerns> can be identified An <Order> begins to 
exist in the memory of the organization when it <Concerns> at least one <Furniture>. Tluit means a minimum 
cardinality of J is necessary. 

Conclusion 

Students, when asked what is an entity or a relationship, use less textbook like explanations and more 
examples. So, the advisor favors the transfer of knowledge. At the end of their curriculum, students have a 
training period within an organization. More than sixty percent asked to use CODAMA during this training 
period. 

References 

Ausubel, DP. (1960). The use of advance organizer in the learning and retention of meaningful verbal 

material. Journal of Educational Psychology, 5 1 . 
Boulet* M.M. (1992). Advisor systems: The development life cycle illustrated by two case studies. Quebec: 

Edition Dwayne. 

Chen, P.P.S. (1976). The entity-relationship model - toward an integrated view of data. ACM Transactions on 
Database Systems. 1,9-36. 

Leshin, C.B., Pollock, J., & Reigeluth, CM. (1992). Instructional design strategies and tactics. Englewood 
Cliffs: Educational Technology Pub. 



177 



1S5 



H a F?*£"' Mm £ Grab l n ^ er V?'. S - ( i?? 4) ' A PP ] y' in & HCI Principles to computer-based higher order learning environments. In M. Orey 
(Ed.). Proceedings of the Thtrty-fifih ADCIS Conference (pp. 178-185). Nashville, Tennessee: Omni Press. 
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Abstract: While computer-based learning has been going on for years in many 
different forms, more recent trends have been towards environments that promote 
higher level learning and that can effectively instruct in more complex subject 
domains. This paper explores the HCI issues particular to computer-based higher 
order learning environments and how these may be addressed via various 
implementations of HCI principles, as well as how general HCI principles can be 
applied in these applications. 

Introduction 

The content of a hypermedia/multimedia application may be right on target for an intended task but, if 
the interface between human and computer is not designed well, then the information and instruction may be 
ineffective or inaccessible. While screen design issues (i.e. placement of text, font sizes) are certainly a 
component of the human computer interface (HCI), this paper discusses HCI issues at a broader level, dealing . 
with concepts such as the "intuitiveness" and "transparency" to improve the clarity, usability, and effectiveness 
of computer systems. 

"Clarity, usability, and effectiveness" are far reaching, fuzzy goals. The domain, therefore, of HCI 
design is not clea*' cut or easily defined and ranges into a variety of issues including the physical arrangement 
and ergonomic configuration of computer systems, user operation of programs, and how the user interacts with 
the content to solve a task or to learn material. In the case of an instructional program, the relationship between 
HCI design and content also affects the instructional design of the program because the interface must not 
interfere with any instructional strategies embedded in the program. Therefore, some of the principles and 
guidelines discussed in this paper go beyond how users interact with the computer and program to include how 
the interface effects the users' interaction with the content in an instructional situation. 

HCI Definition 

Like many concepts, there are a variety of definitions for human computer interface design. In broad 
terms, Carroll (1987) describes the notion of human computer interface as "...computers as experienced and 
manipulated by human users" (p. xx). Reisner (1987) elaborates on this by describing the relationship as two- 
sided though uneven: we are able to predict specifically what the computer will do as the result of certain input; 
however, it is the unpredictable human reaction to the computer that we need to study and plan for in our design. 

Palme (1983) presents an interesting metaphor which describes why HCI design is such an important 
consideration in program design. He contrasts the computer and the rules that create its interface to the rules that 
govern society. There are general rules that govern society behavior that are valid in numerous different 
circumstances. For example, loud noises, coughing, and talking are discouraged at movie theaters, concerts, 
plays, and church services. This is easy to understand and transfer to different situations for flexible human 
beings. The computer, however, requires a more complete and more rigid set of rules than society does, for a 
computer does not readily understand nuances or similarities in situations. The software must define every 
possible reaction for the computer. In society things are looser and there is more inteiprctation of the rules, 
meaning the rules are not always followed exactly the same way every time. This difference in flexibility and 
rigidness is one way of explaining why humans may have trouble dealing with a computer's set of inflexible 
rules, for while the human user perceives the differences inherent in different contexts, the computer only 
understands one reaction or algorithm. This is why the definition of the rules that make up the HCI arc so 
important and complex. 

For purposes of this paper, an HCI is defined as the layer of the software that communicates directly to 
and interacts with users. A sampling of critical components of a HCI includes: 
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screen designs; 

all "messages" (error, status, etc.) to users; 

interactions, flow, or navigation between screens or other various parts of the program; 
inter-relationships between messages within the program. 

Note that this list goes beyond the ergonomic factors such as the physical layout of the software and the 
ways humans access that software (i.e., keyboard, mouse, monitor) through the hardware configuration. An 
interface is interwoven throughout the entire program— it is in the way that the software reacts to users when 
they need help or make errors and in the way the content is presented. Likewise, the principles and design 
guidelines that follow must deal with these diverse aspects of a HCI. 

BASIC HCI PRINCIPLES ~ 
Guideline 1. Conduct an audience analysis to guide the development of the human computer 

interface* 

A central HCI design principle is to analyze the characteristics and needs of the audience (Carroll & 
Rosson 1987- Bailey, 1985). Although instructional programs include an audience analysis as part of any 
instructional design, audience analysis for the HCI differs slightly from audience analysis in instructional design 
(ID). ID learner analysis focuses on the cognitive, affective, and prior learning characteristics of the learners. ID 
analysis looks for characteristics that will facilitate or inhibit learning. Task and performance analyses for HCI 
emphasize analyses of intended usage patterns and technological attitudes and abilities. The following 
guidelines look at how to address these HCI issues. 

Guideline la. Examine the problem the user intends to solve with the software. 

While this guideline does relate to ID's task analysis, one must go a bit further in this analysis for HCI 
design ID always examines the principle subtasks inherent in completing the main task. One factor to consider 
in the HCI analysis is whether any of the subtasks can be performed by the interface rather than the user to help 
the user maintain appropriate cognitive involvement 

For example, if the task is one that users are familiar (e.g. writing a check, solving a common 
mathematical problem, or entering data in a frequently used program) with then the HCI can take advantage of 
commonly known terms and perform mundane steps to enable the users or learners to focus on new steps and 
new learning. That is, users may know how to write a check without a computer, therefore, ma new 
computerized version, the HCI would guide the user through the well known steps using common terminology 
and emphasizing unfamiliar tasks of the computer version. On the other hand, the opposite situauon may be 
true Students may be so engaged in a new, large task (e.g., simulating a science experiment, companng 
geographic information) that they fail to see and adequately perform the subtasks. The interface must ensure 
proper attention to detail and cognitive subtasks necessary to complete the larger task. 

Another aspect a thorough user analysis is to look at users' prior behavior patterns in performing the 
task. Sometimes a new program is buUt to replace an old program. In that case the following questions are 
asked 

Did users perform the task on a computer before? If so, will the new solution be a similar type 
of program or very different (i.e. going from a DOS environment to a Macintosh)? 
What did users like and dislike about the prior program? Were disliked factors a result of 
poor implementation? 

Were users forced to use it against their will thus creating a bad attitude that no matter how 
good the program it would not be appreciated)? 
If software was not used before, these questions won't apply. The following questions should be 
examined for this class of users, but are also useful when the audience has used computers before for the task. 

What aro the users attitudes toward using computers? For instance, have they had or heard of 
experiences where computers have "replaced" people, or had any other experience that may 
cause them to have a negative predisposition to any software introduced. 
Should the program imitate the existing way of accomplishing the task exactly? The wider the 
difference between the computer method and noncomputer method, the flatter and longer the 
learning curve. 

Once designers have collected this information, it can be used to make decisions about not only the 
HCI, but also what else might have to be done in order to achieve success with the product For instance, if there 
is a negative attitude regarding computers, the software may have to be accompanied with assurances I ruin 
some trusted body or individual that this software is not threatening any person's job, or that a learner s prior 
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bad experience with a "drill and practice" program does not necessarily mean they won't succeed with the 
current environment. 

Guideline lb* Examine the users experience level with computers and other relative 

kinds of software packages* 

The users* experience with other software packages tells something about what expectations and what 
metaphor they may bring to the product being designed. If users have experience with command line systems 
and programs (e.g., a DOS or UNIX system where users must know the command to enter to achieve the desired 
action, versus a menu driven system), then a system or program that reiies on icons and menus, while 
supposedly easier to use, may be unfamiliar and difficult to those users. This means that a significant amount of 
education will need to be done before users will adapt the new system and more on-line help may be required. 

An interface can also attempt to relate the current interface's functionality back to a system users arc 
already familiar with. So a help message about pull down menus might define them in terms of a command- 
driven system. An extension of this concept is to actually create an interface that allows multiple modes for 
running the system; that is, there might be an option to emulate a command-line system (or whatever system 
type users are familiar with) rather than using the pull-down menus. If the user community's prior experience is 
not homogeneous, as is likely to be the case, designers may not be able to accommodate all styles wilhin an 
interface Oust as instructors are not able to adapt to all individual learner characteristics), but may need to model 
the interface metaphor as described before, so users may have a better chance of knowing what to expect of the 
interface. 

Guideline lc. Examine user attitudes toward using a computer for the intended task. 

Users bring many attitudes toward computers. Some may see if as a great "tool" that can make them 
more powerful on their job, while others may feel quite threatened The latter group may have heard or 
experienced folks losing their jobs as use of computers increased. Knowing this type of information can affect 
the final product in a couple of ways. First, while designers should always strive for the best, in a case like this, 
being extremely sensitive to a friendly, easy-to-use interface that clearly shows the benefits to users becomes 
even more important Even sensitive interface design, though, may not totally compensate for these pre-existing 
attitudes, so a design team may want to suggest that the product introduction be accompanied with written or 
verbal materials that attempt to ease users* concerns. 

On the other hand, if users bring a positive attitude toward the system, then they ought to be able to 
find the kind of functionality they expect in the product. The interface should make this functionality obvious 
and simple so it can capitalize on users' enthusiasm. The rub for designers comes when trying to adapt a system 
to work well for both users looking for lots of functionality and those that are more timid with the product. 
Some suggestions for scaffolding "progressive turnover" of functionality to users as their expertise increases are 
discussed later. 

Section Summary 

To summarize how HCI's audience analysis principle should be factored into instructional design: 

• Task and front end analyses should be expanded beyond learner characteristics to include user 
attitudes and task accomplishment patterns. 

• All stages of design and implementation should balance and measure themselves against the needs of 
users in terms of program operation and program purposes. 

• When in doubt or when other principles or heuristics are in conflict always turn back to users' needs 
and characteristics and make decisions based on them. 

PRODUCTION DIAS 

Guideline 2. Design an interface that encourages exploration and makes learning 

efficiency a rewarding experience. 

Production bias (Carroll & Rosson, 1987) means that users are interested in completing their target 
tasks, or in other words they are "biased" in their use of an interface toward getting their jobs done. They are 
intent on being productive. This isn't a bad tendency if the path they take to accomplish their task is the most 
efficiently designed path. However, using the computer efficiently or as the program designers intended is often 
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of little consequence to users. Their production bias motivates them to finish the task any way they can- 
efficient or not. This bias discourages users from taking time to do any system exploration, experimentation, 
or documentation reading, because anything not directly associated with getting the job done seems a deviation 
from their goal and a waste of time. Users may desire a more efficient way to perform a task and may even 
suspect that such a method exists, but the desire for task completion will outweigh the desire to find a more 
efficient alternative. 

While knowing users 1 degree of production bias isn't totally possible, there are a few things to consider 
that may be helpful in both estimating the bias and determining how to deal with it Once again these come back 
to the importance of a thorough audience analysis. A design team may try to determine the users' prior "patterns 
of use" with previous pieces of software; particularly patterns associated with learning new functionality or 
willingness to conduct system exploration. In many programs it may be possible to collect evidence in 
frequency of use of help facilities or training and documentation. 

If users don't use these help facilities, then find out why. Is it that they have tried and they have been 
frustrated by not finding what they need, or is it that they just haven't tried? In either case, the implication for 
the HCI may be to couple the interface with other aspects of the product thai encourage exploration and new 
learning. For instance, a button labeled "Catalog" may result in a pop-up window that says 



"Enter the name of the new catalog* • 

For more information about what catalogs are and how they are used, press the help button below. 1 



Production bias also effects the instructional design of the program. For example, students may want to 
finish the program as quickly as possible to get back to "real" work activities (or recess). Extrinsically mod vaied 
students tend to do the minimum to finish an assignment (Keller, 1983). Their level of learning is not the goal, 
but simply assignment completion. The strategies the learner uses to learn the material are a matter of student 
choice, but can be effected by careful instructional design. However, caref ul HCI design can help make sure that 
the strategies are implemented. Handling the production bias may be a simple matter of limiting user options. If 
it is important for users to read and study a particular sequence, the interface and strategy can require a 
thoughtful response before continuing and prevent users from "escaping" to another part of the program before 
the section is complete or before the objective is achieved. Similar to interface design, the instructional design 
of the program tries to develop strategies for the instructional interface to motivate learners to go beyond the 
minimal target goal. If high level learning is a goal of the program then the program interface should support 
this via exploration encouraging methods such as inquiry teaching (Collins & Stevens, 1983). 

A second instructional design factor affected by production bias is the opposite of the previous 
example. Instead of lightly covering material, students try to memorize everything they encounter for they 
believe that is th*,ir task. For example, programs that implement cognitive flexibility theory, such as KANE 
(Spiro & Jehng, i990) require the learner to dynamically link back and forth between a number of nodes 
examining many perspectives on the complex topic (KANE explores the motivations of the character Kane in 
the movie Citizen Kane). The learners may be so unaccustomed to this mode of learning that they assume their 
task is to learn (read or memorize) everything encountered in the system and not use the interconnections 
between nodes to construct their own views as the program intended. So, their production bias leads them to 
accomplish the wrong task. 

Guideline 2a. Given that production bsas leads users away from system exploration 

and learning, make efficient task completion obvious within the user 
interface* 

Realizing that users may perform shortcuts or, conversely, accomplish a simple task with an 
unnecessarily complex routine, HCI designers need to strive to make efficient task accomplishment obvious 
within the user interface. In other words, the interface should lead users to the most efficient ways to accomplish 
a task. In Desert Shield , for example, information access is the prime purpose. Therefore, the interface design 
focuses on letting users access information in multiple ways: from the menu bar, a main table of contents, and 
within fields on the screens. The designer's goal was to provide direct access (one mouse click) whenever 
possible. 

Another way of accomplishing this is to utilize a functional organization of on-line help rather Uuin (or 
in addition to) a topical organization. For example, users may need help on step five of a process, so the 
interface shouldn't force them to look at one through lour. Desert Shield uses a functional system of on Imo 
help by letting users access a help topic for a menu or button choice directly by holding the option key ikmn 
when clicking. Again, this gives direct help without taking the user through a long set of hierarchical sckyhv 
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Guideline 2b, Describe task functions, buttons, and menu options using words, 

phrases, and icons that are familiar and have meaning to the target 
audience* 

A great deal of good interface design is in the language used for describing objects and tasks. For 
example, a table of contents or index entry should read Moving Text rather than Steps in Cut and Paste. 
"Moving text" is the task users try to accomplish; "cut and paste" are names for ' ! ie operations defined by the 
system developers. The names may be descriptive, but they are not as descriptive as what may be familiar to 
users. 

The notion of simple and direct naming should also apply to the design of icons. Graphics-based 
interfaces like Windows and Macintosh foster the development of icons instead of words to label buttons and 
functions. Although a lot of thought goes into these icons, tliey are often a mystery to common usere. All words, 
phrases, and icons need to be tested in a formative evaluation process. 

Section Summary 

The production bias says users are interested in task completion. Their interest in completion is so 
strong they may not take the time to discover the most efficient way to accomplish their task. While there is 
probably no way to eliminate this bias, instruction or a HCI can try a couple of strategies. Functionality that you 
want users to take advantage of needs to be clearly and easily available. Users shouldn't have to look far. 
Additionally, designers should try to find out how much the production bias is in effect by looking at how users 
have learned new concepts and methods in the past. 

ASSIMILATION BIAS 

Guideline 3. Correct any misconceptions or inappropriate knowledge brought to a 

ne w way of accomplishing a task from prior experience and learning. 

Assimilation bias (Carroll & Rosson, 1987) states that users 1 previous knowledge base may obscure 
new learning. It is similar to the production bias described above in that users end up sticking with the way that 
they are familiar with to get the job done rather than exploring or looking for more efficient method. . In 
assimilation bias, users are led to an inefficient way of task completion because of preconceptions from prior 
learning. Users are comfortable with their previous way of accomplishing a task and resist a change to another 
method. They arc reluctant to assimilate the new methodology into their working or study habits. 

Assimilation bias is evident in HCI design when the computer program operation may be so new and 
different that users seek a familiar sequence and methodology to accomplish their task. User-familiar items can 
be defined by prior knowledge of and experience with other systems or can be learned from a co-worker who, 
even though a novice also, may be perceived an expert in comparison. In this way, system misinformation and 
incomplete information spreads among users. From this incomplete information, users may also formulate 
theories and rules about the rest of the system which may be incorrect 

Guideline 3a. Write informative error messages to tell a user why something went 

wrong, not just that something went wrong. 

Inappropriate prior knowledge and its consequences must be discovered and remedied. Much of this 
must be dealt with in the original planning and formative evaluation of the interface, for the HCI tends to remain 
static once implemented. In an interface, a misconception generally takes the form of an incorrect system view. 
In most systems, the only opportunity there is to correct such views is when an error occurs. Therefore 
interface designers must not only consider designing clear error messages, but also suggest ways to correct not 
only the error at hand, but the underlying misconception as well. 

Guideline 3b. Design the interface so users can't "hurt" anything, or have the 

perception that they have hurt something. 

This guideline can mitigate the effects of the assimilation bias. If users believe that they "hurt** 
something, they may become shy with the system and become reluctant to try new things, thus exacerbating 
both production and assimilation biases. This guideline is probably more critical as applied to the novice user 
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who approaches the computer with some trepidation. A significant factor in implementing this guideline is a 
thorough system evaluation . ■« • • . A u t 

A second way to help encourage users to learn different rouunes is to build in an undo and go back 
functions. Unfortunately, these commands may be difficult and expensive to implement as they require the 
system to save snapshots before and after every given command. Less expensive attempts at user safety include 
warning messages when the user is about to do something with sweeping consequences, such as removing the 
contents of an entire directory/folder or not saving the results of a two hour CAI session. 

Section Summary 

The assimilation bias states that user's prior knowledge may obscure new learning. The functional ^ramifications 
are similar to the production bias in that users may not use the interface's full functionality but rather suck to 
tasks they know. Designers can use the following strategies to counter-balance the effects of the assimilation 
bias. 

user error messages to correct/modify incorrect system views, 

correct errors as soon as possible so incorrect system models are not cognitively strengthened 
inadvertently,^ ^ impervious t0 harm so if learners do explore, they will not have the perception they 

have damaged anything; this friendly exploratory environment may lead to further exploration and discover of 
new interface functionality. 

INTUITIVENESS _ 
Guideline 4. Design an interface that is intuitive, logical, transparent, and direct to 

users. 

To be intuitive in the context of HCI design, the interface must adapt to the reality of how users think 
versus expecting users to adjust their thinking schemata and strategies to an arbitrarily designed interface. If an 
interface is intuitive, the interface falls easily within the expectations and experience of the users, therefore the 
users will not have to make much of a leap to place their cognitive processes about the problem they seek to 
solve within the confines of the system. The normal way they approach the target task should be readily 
represented within the system's interface, , . 

Concepts related to intuitiveness include transparency and directness. Designing a transparent system 
means that program operation doesn't interfere with users performing program tasks. Transparency in an 
interface helps create a system that is easy and non-intimidating to use. A transparent interface takes little 
cognitive overhead and puts few, if any, demands on users. Icons, pictorial representations of interface 
functionality, are a potential example of interface transparency. A small picture of a computer printer on a 
button is thought to be transparent for most users because it is an easy leap to understand the function the button 
activates. 

An interface that is direct helps users believe that they are engaged in accomplishing their goal. An 
indirect system serves as an obstacle to accomplishing goals (Hutchins, Hollan, & Norman 1985). For example, 
a word processor that is direct helps users believe they are writing a story and not using a word processor to 
write a story. A CAI program that is direct helps students believe that they are learning valuable informauon 
rather than operating a computer program about information. 

Welsh, Murphy, Duffy & Goodmm(1993) found that a non-intuitive, non-direct way of labeling 
hypertext links had negative effects on how much students accessed the information via that link. Once students 
chose to access the link, they received a sub-menu of choices from which they had to choose again. The effect 
of this indirection was students simply avoided accessing the links at all. The same experiment showed direct 
link accessing methods where information was presented as soon as the link was chosen were used more 

frequently. . . . TT 

Another way of describing the principle of intuitiveness is via metaphor matching. Users come to an 
interface with a metaphor about how the program functions. A metaphor creates a set of expectations among 
users about how difficult a task will be a'd how they will perform that task. For example, when Apple 
Computer introduced the Macintosh, they also introduced a new metaphor for computer operauon, the desktop. 
Users are encouraged to view the computer as a desktop with a series of folders to organize their work. If the 
interface operates consistently with the users' metaphor, then the interface feels intuitive or natural. If not, users 
must adjust their metaphor or expectation in order to use the system most effectively. Then the interface is not 
intuitive, transparent, or direct. 
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For example, frequent users of command line driven systems (a "command line driven" system takes 
m0St ° f * d& USCr inpul via 0001111811(15 computer understands typed in via a keyboard) , like UNIX® 
and DOS®, may expect computer interactions to consist of typing a command at a prompt and then getting a 
response. When command line users are fust exposed to a system using the desktop metaphor (e.g., Windows or 
Macintosh), they find that their expectations are not met Users have to adjust their expectations in order to use 
the desktop system without having to translate the commands they would have normally typed into the 
equivalent desktop inputs. So what was intended to be a veiy easy interface to adapt to, turns out, at least a 
subset of users, to be quite a leap. 

As effective as a metaphor is in tapping into something familiar and easy to grasp for the users, it can 
be overused and actually decrease transparency. Jonassen (1989) cites an example of using multiple windows in 
an interface. Placing programs and files in windows on the screen can be an intuitive interface device because 
they extend the concept of the single view of an application or task to multiple views. This may be intuitive to a 
point, but when too many windows are available on the screen they have the opposite effect. Users can 
only juggte effectively a few windows at once, just as a person can only keep a few items in short-term memory. 
So, once the number of windows gets beyond a threshold, depending on each user and the context in which they 
are using the interface, they become a hindrance as the user to has to concentrate on the windows rather than the 
tasks they contain. 

The interlace should also help make the instruction intuitive. Instructional design theories such as 
cognitive apprenticeship (Collins, 1991) also have goals of intuitiveness and directness. These instructional 
methods strive to immerse learners in a real-world learning experience to make the learning effort intuitive 
while focusing on a realistic task. 

Guideline 4a* Keep the interface simple and basic with a limited number of options. 

Avoid designing an interface that is too feature-rich, or too busy. There is a basic set of functionality • 
users need in a software package in order for it to provide an enhanced way of accomplishing the desired task 
While it is true that multiple ways to achieve a task may help an interface work for novices and experts, too 
many options may confuse users (especially the novice) and actually detract from the interfaced usability. From 
a cognitive processing point of view, users must chunk away all the options in short term memory, and if there 
are too many of them, their own processing is bogged down causing confusion and reduced 
interface transparency (Andre & Phye, 1986). 

For example, the development of software for phone switches (which these days are complex 
computers), results in the conclusion that most users don't want or need the feature richness provided in these 
systems. The multitude of ways to answer calls and place them on hold, and many of the features provided for 
accomplishing those functions often go unnoticed and unused. Users evidently find one way or possibly two 
ways that are intuitive to them to accomplish the task, such as answering the call, and don't search any further 
for others unless their needs change and that need outweighs the effort of looking for and learning a new feature 
So users, in this case, actually, provide their own simplicity in an interface that offers too many options. 

Guideline 4b. Balance the locus of control between the interface and users* 

Designers should build an interface that encourages users to interact with the system in substantive 
ways that reflect the actual nature of the task. For example, some of the older, line-oriented word processing 
programs had such severe limitations on how to represent information that users had to format text in their 
minds rather than on the screen to meet the text editor's demands. The antithesis of these editors are today's 
WYSIWYG (what you see is what you get) editors where users have almost total control over the final 
appearance of the text. Although, in the first case, users may eventually memorize all of the editor's interface 
rules, these rules are not substantive parts of the actual task. In the second case, the interface design has adapted 
to the users to give them more control over the job. This frees users to concentrate more on the content of their 
text instead of concentrating on how to use the tool. 

Section Summary 

Ironically, there is no intuitive way for a designer to necessarily create an intuitive or transparent 
interface. However, designers need to considci their audience and design interface attributes that fit as easily as 
possible in to the users' cognitive structures. Keeping the interface relatively simple and balancing control 
between the interface and users arc ways designers may promote interface intuitiveness. 
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Intuitivcncss, directness, and transparency are important elements of a successful interface. In order to 
design these characteristics into the interface, designers must understand the characteristics of the intended users 
and the intended use of the system. This level of understanding includes the expectations and metaphors users 
bring to the program, as well as how to best actively engage the learner in the completion of the tasks they arc 
working on. In i am then, the interface must allow users to concentrate on their target task instead of the 
interface itself. If an interface can accomplish this, it has achieved an important level of intuitivcncss, directness, 
and transparency. 



CONCLUSION 



This paper has discussed the principles and guidelines for HCI design with emphasis on these 
principles for hypermedia and multimedia design. Principles discussed included 

• doing a thorough audience analysis to determine both users abilities and prior experience with 
computers and interfaces as well as attitudes, 

• accommodating the users production bias by encouraging exploration within the interface, 

• accommodating users natural assimilation bias with an interface that corrects inappropriate prior 
knowledge that may limit users abilities within the interface, 

designing an interface to be intuitive, logical and clear to users, 

• and building an interface that supports adapting and growing with the users rather than the other way 
around. 

There is no doubt that these concepts are difficUt to grasp and even more difficult to implement as a 
designer. However the importance of a well-designed HCI can not be over stated. The HCI is the primary way 
users interact with a learning environment and if they find it difficult to use, their learning may be compromised. 
Conversely, any design effort that results in an HCI that allows users to focus on the learning task at hand and 
enhance their abilities in that arena, while it may not be overtly acknowledged by users, will pay off with users 
having a more satisfactory experience with the learning environment. 
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Introduction 

The field of computer-based instruction (CBI) has grown by staggering leaps and bounds in ihc past ten 
years. Additionally how programs are controlled and what they look like has changed. Early CBI programs 
were typically tutorials which were linear in nature and offered few choices beyond "Press the Space Bar to 
Continue." Today's programs are a fusion of buttons, pull down menus, cascading windows, and overlapping 
control panels. While these technological advancements make programs more powerful and impressive, the 
technological breakthroughs of today and tomorrow must be harnessed in order to make these complex new 
programs effective tools that can promote human learning. 

The purpose of this paper is to examine how metacognitive theory can be integrated into the design of the 
computer based learning environments through the user interface. A description of interface design and 
metacognitive theory will be presented, and, a list of organizing questions will be used to demonstrate how 
metacognition can be integrated into the design of a user interface. 

Interface design 

The interface of a computer-based instruction (CBI ) program can be likened to a control panel from which 
users access information in an oftentimes sophisticated and complicated piece of software. Users must be able 
to control course presentation, access the elaboration of unknown terms, initiate animations, manipulate objects 
such as multiple windows, navigate to test items, make responses to questions, be able to ascertain where they 
arc in the program, and know how well they are progressing. All of these activities associated with CBI 
contribute to the interaction style of the program. The interaction style of a program depends on the number and 
kind of tasks learners arc required to perform in a program, and the number of choices available to them. Once 
an interaction sty!etfas been determined, the designer can then begin determining how the choices required by the 
users can be physically laid out Interaction styles can be manifested within the interface as physical 
representations of choice. Interface design elements as menus, buttons, multiple windows, help and 
navigational aids, and response types (Jones, 1993) represent these choices. The design of the user interface 
serves as a cognitive dashboard from which users can control the program and monitor their progress towards the 
end of learning the information contained in the program. In short, the interface of the program often represents 
the implementation of the instructional strategy of the program. 

Metacognition 

Ideas and theories stemming from the current popularit> of cognitive psychology continue to influence the 
theories of instructional design and the development of computer-based learning environments. Research on the 
use of complex skills has led cognitive scientists to include cognitive strategies as a type of skill that is used to 
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solve difficult problems. This special type of intellectual skill is an "internal process by which learners select 
and modify their ways of attending, learning, remembering, and thinking" (Gagne, Briggs, & Wager; 1988). 

Cognitive strategj.es include rehearsal strategies, elaboration strategies, organization strategies, affective 
strategies, and comprehension monitoring strategics. These strategies are cognitive events that describe the way 
in which one is processing information. The design of a computer-based learning environment should lake into 
consideration the users ability to engage in, preference for, and appropriateness of various cognitive strategies. 

Metacognition is a type of cognitive strategy that is said to have executive control over other cognitive 
strategies. In the context of learning through a computer-based learning environment, metacognition refers to 
the cognitive activities of a user to monitor, regulate, and orchestrate learning processes (Flavcll, 1979). Figure 
1 describes components of metacognition. 



Metacognition 




Control of 




Monitoring of 






Cognitive Processes 




Cognitive Processes 






• Strategy Selection 




• Goal Setting 






♦ Attention 




♦ Goal Checking 











Figure 1. Components of metacognition 



Four individual strategies within metacognition include strategy selection, attention, goal setting, and goal 
checking. These strategies can be grouped into two major categories: 1) control processes and 2) monitoring 
processes. The two categories of mctacognitive processes are described below. 

Control Processes 

As the executive controller of cognitive processes, metacognition selects the approoriate strategy for the 
task at hand. The selection of a cognitive strategy depends upon the individual's understanding of the current 
problem or cognitive situation. Personal experiences in solving similar tasks and using various strategics will 
affect the selection of a cognitive strategy. For a learner, this may involve the selection of rote rehearsal as a 
strategy for learning particular facts such as a list of vocabulary words. 

An individual can also choose to attend to particular cognitive strategics. This strategy, attention, is 
important in following-through, completing and correctly performing the steps of subordinate cognitive 
strategics. In other words, this mctacognitive strategy maintains one's interest in using a particular strategy 
after that strategy has been selected. 

Monitoring Processes 

Cognitive processes such as learning and problem-solving often begin with the identification of a goal. In 
learning, this might be an understanding of a particular topic; in problem-solving, the goal would be to find a 
solution (or the best solution) to the problem. Goal setting guides the cognitive strategics in a certain direction 
Goal checking arc those monitoring processes that check to see if the goal has been accomplished, or if the 
selected strategy is working as expected. This monitoring process is active throughout an activity, constantly 
evaluating the success of other processes. If a cognitive strategy appears not to be working, an alternative may 
then be selected. Various prescriptions for instructional development proposed by Redding (1990) arc primarily 
concerned with giving mctacognitive guidance in problem-solving situations. This guidance includes many 
familiar methods such as advance organizers, graphic representations of problems, and hierarchical knowledge 
structures. These instructional methods should be used to aid the novice in developing an expert's awareness of 
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problem space. "The most effective way to turn a 'poor' novice problem solver into a "good" novice problem 
skills" ^eddm^ljgo) 1 ' 1216 dCVel ° PmCnt of ex P erti8e » h 10 leach student P«*lem space representational 

Metacognition is the "management" of thought processes as one leams and solves problems In the 
context of a computer-based learning environment, learners are presented with large amounts of information, and 
asked to sift and sort through that information to solve a particular problem, or learn about a particular topic In 
order to assist the management of this information, the interface should provide users with relevant information 
about the program, how to use the program, and how well they are doing in the program. To effectively use 
these programs, users will need information, and users will have questions. We have chosen three organizing 
questions that users will have when confronted with a computer-based learning environment, and provide 
suggestions for how a designer might provide users with relevant information: What is it? How do I use it' 
and What do I know? In a previous study. Jones (1993) sets forth a list of concepts of user interface design and 
corresponding guidelines for implementing those concepts. Using the organizing questions presented in this 
paper, we will attempt to provide guidelines for designing user interfaces which can answer these questions for 

What is it? 

Learners select and use programs based upon unfulfilled needs or particular interests. If the program is 
selected as a learning activity, then a goal has been established. The use of a program often results from a 
decision to reach a goal through a particular strategy. Goal-setting and strategy-selection are key elements in the 
personal irianagement of cognitive processes. Users ask them-lves: "Does this program fulfill my need, and 
will I be able to use the program efficiently and effectively?" To assist the learner with this management it is 
the responsibility of the program to communicate the content and scope of the environment. In the event'that a 
prospective learner has an ill-defined goal, it is the responsibility of the program to provide suggestions 

What is it. our first organizing question, refers to the initial way that the program refers to what the 
program is all about. Users will need information about the subject of the program, the s-;ope of the 
information contained in the program, and the type of information contained in the program. This information 
is needed not only when users start the program, but throughout the program as well. In a complex computer- 
based learning environment^ users may have questions about the scope of the program throughout the entire 

!* COnCeplS ° f interaCC dcsign (Jones> I993) address to Question of What is if? as it applies to the 
providing metacogni'.ve support within the user interface: Metaphors. Browsing and Closure 

One way that designers can answer the question of "What is it?" for users is through the use of a program 
metaphor or organizing theme. Metaphors help users define what information is co-.tained in a program by 
relating it to a known function or process taken from an area or discipline familiar to the users (Jones 1989) 
Program metaphors such as books, book shelves, space exploration, or buildings with different rooms' help 
users to organize the program's content and the access to the content contained in the program. The following 
guidelines can help designers provide users with efficient program metaphors. b 

1 . Use a metaphor or theme for the program. 

Not every program needs a metaphor. Not all programs can support a metaphor. Study the content 
carefully and decide what the program is intended to do. Providing users with a theme can be more 
helpful than a forced or inappropriate metaphor. 

2. Make the metaphor obvious to users. 

Try prototyping the metaphor and its controls and testing it with potential users. Be careful of 
unsung your instincts on whether of not the metaphor is obvious, and get the opinions of people who 
are not familiar with the program. 1 

3. Make the metaphor applicable to the program's content. 

If a metaphor can be used, use a metaphor that reflects the program's content. Users should not have 
2 r£?.h ,ii nCanil !r 8 3 meta P hor mi mc comcnt of thc Program. Some metaphors may be intended to 
reflect the theme of a program and not necessarily the program's content. For example, a flight simulator 
would probably use the metaphor of a cockpit to show users how a plane works. However a program on 
different airline regulations might use a metaphor of a resource manual or book. 

One way that users often acquaint themselves with the informauon contained in the program is through the 
process of browsing for information. Browsing allows for the flexible exploration of the content of me program 
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through a variety of controls (Jones, 1989; Laurel, Oren, & Don, 1992). Browsing can be done topically by 
providing users with a list of the topics covered in the program through the use of a menu. Once a topic is 
selected, users can use methods such as clicking on right and left arrows to access related or extended material. 
While browsing should be flexible and exploratory, it should not be indiscriminate or uncontrolled. Users need 
to be able explore the program for new information, know where they found it, and be able to find it again. The 
following list of guidelines can help designers provide for effective and innovative methods of browsing in their 
programs. 

1. Provide selectable areas to allow users to access information. 

Some possible selectable areas to consider are buttons and hot text within a text field. The location 
of these elements on the screen will depend on the available screen real estate and the function of the 
selectable areas. It is recommended that the placement of selectable areas be tested with users to find out 
what is the optimal location for them. The selectable area will be a control element for users to access 
information. The control chosen will depend on the task to be done. Be consistent in implementing 
particular controls for particular functions. 

2. Allow users to access information in a user-determined order. 

This may be done through topic indexes of all of the information available in the program, or 
through the use of different types of menus. Another technique to consider is allowing for user-entered 
search terms. Exploration should be flexible, and the controls for accessing information should reflect 
flexibility. 

3. Provide maps so thai users can find where they are and allow provisions to jump to other information 
of interest from the map. 

Because the content of computer-based learning environments tends to be complex, using visual or 
iconographic maps may be too difficult to include and too confusing for users to understand. What we 
now consider as maps may have to change drastically. Text based indexes, outlines, and tables of content 
may be considered as alternatives to maps. 

Another method that designers can use to help users answer the question of "What is it" is by incorporating 
closure (Jones, 1989) into the program. Providing users with closure can help organize the information in the 
program into manageable segments so that users are not overwhelmed by the amount of information contained 
in the program. Organizing information requires that methods be used to allow users the ability to access 
information in a controllable fashion. The following guidelines can help designers provide closure within a 
computer-based learning environment. 

1. Provide users with information that lets them know that they are making progress. 

Because the information in computer-based learning environments is not organized sequentially, there 
is no determined order that users must follow through a program. Consequently users may feel that they 
are working in a program without making progress. Some techniques that may be considered to give 
users a sense of accomplishment are path history mechanisms that tell users what information they have 
seen, or visual cues that indicate progression. Another technique would be to break the program up into 
chunks that may give learners a feeling of accomplishment. 

2. Arrange information in a non-threatening manner so that users are not overwhelmed by the amount of 
information contained in a program. 

To accomplish this consider setting up information with an overview of a topic that acts as a top 
layer of information. As users need more information they can move progressively deeper through the 
layers of information. Moving through the layers of information could be done through the use of pop- 
up menus, buttons, or hot text, 

3. Provide visual effects to give users visual feedback that their choices have been made and registered by 
the program. 

Buttons, icons, and menus can be highlighted or animated to show users that a choice has been made. 
Keep the highlighting or animation simple. The duration of a highlight or animation should be long 
enough to be registered visually by the users, but short enough so that users arc not wailing for an 
animation to be over so that they can get to the information they want. 
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Visual effects, such as wipes, fades, and zooms may be used to indicate access to a particular piece of 
information. The use of these visual effects should be consistent Do not use them simply because thev 
are available, but rather use them to indicate a particular action of the program. Additionally, be 
consistent in the use of a visual effect. If wipes are used when clicking on a right arrow, use' them 
throughout the program. If zoom outs are used when clicking on a menu item, then use zoom ins when 
returning to the menu. Above all, make the visual effect have meaning and be consistent with its use 
throughout the program. 

How do I use it? 

Managing cognitive processes involves the constant selection of learning strategies. Once a goal has been 
set, learners will ask: "How do I achieve my goal?" One of the benefits of large multimedia environments is the 
availability of numerous learning strategies. Some programs offer a seemingly endless variety of accessibility 
and interactivity opUons. Other programs present a select few. In assisUng the learner with the process of 
strategy-selection it is imperative for the program to communicate options and recommend methods. 

Invariably, when faced with a complex computer based learning environment, users will need to know how 
to use the program. For designers of computer based learning environments, it appears fairly clear- read the 
screen, highlight information, click on buttons, and pull down menus to go where you want to go However 
to novice users, these are not natural activities. Consequently, designers need to make the functions of the 
program not only easy to use, but easy to learn. Answering the question "How do I use it?" is imperative for 
novice users. Three concepts of interface design (Jones, 1993) address this issue as it applies to providin K 
metacognitive support in computer-based learning environments: Information access, Interactive Tools lor 
Interactive Tasks, and Unfamiliar Territory. 

Implementation of the concept of information access (Laurel, Oren, & Don, 1992) provides users with the 
controls to conduct deliberate searches for information to fill a particular need. These information needs could be 
to search for an answer to a specific question, or to find and organize information for a report or presentation 
Informauon access is sometimes manifested through user entered search terms, or by providing users with an 
accessible index to all of the information contained in the program. Some guidelines for providinu for 
information access follow. 

1. Allow for guided searches for specific information. 

This may be done through the use of indexed lists of all the information contained in the program or 
by letting users type in search terms. Additionally, designers may consider using some type of path for 
users to take through the program when searching for certain types of information. For example, if a 
program is on baseball, designers may provide predetermined paths such as "the evolution of the 
designated hitter rule," or "Joe Pcpitone's career from 1965 to 1970." 

2. Allow users to search for information across different media types. 

This guideline may be one that is not employable at the moment It accounts for idea or concept 
searching for informauon within different media types. While interactive video allows for access of 
designer determined sections, all video needs to be searchable for single words, phrases or ideas. Video 
needs to be cross referenced with other types of information in a user determined manner 

Searching for video and audio by key words is technically difficult This type of searching may be 
accomplished through the use of verbatim transcripts of audio and video segments that arc arranged by 
different ideas and concepts. These ideas and concepts could then be active, as hot text or hypertext links 
all: wing users to select them to go to the accompanying media. 

The concept of interactive tools for interactive tasks (Reingold, 1990) suggests that tools for interacting 
with the information contained in the program need to be appropriate to the task, and capable of supporting the 
program theme. The lorm of the control needs to reflect its function, and the function of the control needs to be 
consistent with the program theme. For example, if the program is intended to simulate the control of an 
airplane, then the buttons and other maniputatablc areas on the screen need to look like controls found in an 
airplane. Designers arc encouraged to "push the envelope" of what authoring tools can do in order to develop 
better and more efficient interactive tools. Even if you don't know what to call it, you can still develop it The 
following guideline can help one push the envelope. ' 

1. Provide navigational tools and interaction styles that allow users to interact with the information 
contained in the program in a manner that is consistent with th°. metaphor. 
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If the metaphor of your program is that of a book, then tools such as a table of contents, pages, and 
chapters could be used. Select tools that are consistent with the metaphor. Design icons that reflect the 
theme or metaphor of the program. As in the building of a Frank Lloyd Wright home, the form of a 
control should reflect its function. Additionally, the controls should be integrated harmoniously into the 
metaphor of the program. 

One of the problems that users face when using a computer-based learning environment is in navigating 
through the unfamiliar territory (Nicol, 1990) that is the problem space. Manifestations of the concept of 
unfe* iliar territory provide users with visual or verbal cues that help them begin to learn their way around a 
n . ning environment. Overviews, menus, icons, or other interface design elements within the program 
should serve as advance organizers for information contained in the program. Users new to a program need to 
have methods built into the program to serve as organizers for the information in the program. The following 
guidelines may help designers build effective organizers into a program. 

1. Provide cues such as maps and menus as advance organizers that help users conceptualize the 
organization of the information in the program. 

Advance organizers can help users know what information is available in the program so that they 
have some direction about where they need to go in the program. 

2. Use a fixed format of frames to keep the information at the same place on screens. 

If more than one format for screens is needed, keep the formal constant for individual sections, 

3. Provide users with program orientation. 

Video overviews, overviews in the program documentation, and overviews in the program can help 
orient users to the amount and type of information in the program. 

4. Provide users with information to let them know where they are in the program. 

Section titles, standard borders for specific types of information, constant backgrounds for a 
particular section, and even icons help users can let users know where they are and what they can expect. 
Additionally, using different shapes can let users know where they arc in the program. 

5. Use location indicators and progress reports to let users know where they are, where tiiey need to 
and how long it will take them to get there. 

Location indicators and progress reports let users know where they are and where they need to go. 
These location indicators may be likened to the signposts or mile markers one would see while driving on 
a highway that tell you what road you are on and how far it is to the next town. These are particularly 
useful when users are trying to solve a problem or answer a question provided for by the program. 
Location indicators and progress reports may be represented through text messages, graphics, or pop-up 
windows. 

What do I know? 

As we have indicated throughout this paper, metacognition, or the management of cognitive processes, 
involves goal-setting, sfategy-sclcction, attention, and goal-checking. The question of "What do I know?" 
addresses the issue of goal-checking. Here, the learner will evaluate the effectiveness of the selected strategics. 
The learner will ask: "Have I learned something? Have I achieved my goal? Were the strategics I chose 
successful?" The user interface of a program can support this knowledge in a variety of ways. 

We do not present a formal list of guidelines for the question of "What do I know?**; that information is 
beyond the scope of the original study on which the guidelines are based (Jones, 1993). However, wc do present 
the following issues associated with the question of "What do I know?** as a suggestion for where future research 
on the integration of interface design and metacognition might be focused. 

The user interface can indicate the content covered in the program through the use of advance organizers 
such as menus. It can indicate the content covered by users through selection indicators, such as highlighting 
choices, placing check marks on menu items after they have been selected, or any like mechanism to indicate lo 
users that they have been through the material. One traditional problem associated with marking selected 
sections is that indicating that a section has been selected docs not mean that the users have studied all of the 
information in that section. Accessing information does not automatically mean that the information has been 
studied and processed by the users. 
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Providing for "self-evaluation" methods within the computer-based learning environment is also a way for 
users to check what they know. Long represented in CBi as testing exercises such as "Check Yourself or 
"Final Tests," these types of evaluations make it possible for users to gain information on how much they have 
learned through the use of the program. In a linear tutor.il, designers can provide users with a standard set of 
questions which can be used to check one's progress and performance. However in a complex environment such 
as a computer-based learning environment where users define their own problems, individual users may take 
different paths and find and study different types of information. Providing for "self-evaluations" in these types 
of environments may require different types of exercises than we are currently familiar with. And in ihe design 
of the computer-based learning environment, this can translate into a significant programming task. Once 
again, this an area of interface design where designers and developers need to work together to push the 
proverbial envelope of what can be done. 

Regardless of what type of testing or evaluation exercises are used, users will need notification of 
correctness to various responses. Once again, tradition and tools have often dictated the use of easy to define and 
check multiple choice questions. However, to provide users with realistic self-evaluation mechanisms, it will 
be necessary to design and develop new and different testing strategies. We recommend this as an area of 
continued research and development. 

Conclusion 

The area of user interface design in CBI and computer based learning environments is rife with opportunities 
for further study. As technology changes it becomes increasingly important for designers to put together 
methods for users to access information. There are many people involved in the design of user interfaces. 
Artists, philosophers, computer scientists, writers, and instructional technologists (Jones, 1993; Laurel, 1990) 
are but a few of the types of people working to make user interfaces better. Regardless of who is involved in 
the design of the user interface, it should never be forgotten that the puipose of the interface is to make the 
information contained in the program available to users. Further study on how users use programs, and what 
types of information are required by users can only make interfaces more effective, more efficient, and more 
engaging. 
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Lets Learn Ergonomics 

ALAN WIER and TONG-HO LEE 
Department of Industrial Design 
380 Hopkins Hall, The Ohio State University, Columbus, OH 43210 
614292.7311 email: awier@magnus.acs.ohio-state.edu 



Abstract: Let's Learn Ergonomics is courseware developed originally as a Masters 
Project by Jong Ho Lee (M.A. 1992) of the Department of Industrial Design at The 
Ohio State University. Asst Prof. Alan Wier, the major faculty advisor on the project 
has continued to refine the courseware based on two formative evaluations. The 
Let's Learn Ergonomics courseware was created using Authorware Professional (v. 
1.7 on the Macintosh platform). The Masters Project version was completed in May, 
1992. 

The Let's Learn Ergonomics courseware was designed to teach undergraduate 
industrial design students the basic concepts of the ergonomic design process. The 
courseware development process was: preliminary investigation, problem 
identification, project goal establishment, instructional system design, program 
development and formative evaluation. 

The following paper is a summary, prepared by Alan Wier, of Jong-Ho Lee's Master's Candidate 
Thesis. 

Preliminary Investigation 

The preliminary investigation was conducted between September 1991 and March 1992. It reviewed 
and analyzed existing ergonomic data sources and existing teaching methods and techniques to 
identify strengths and weaknesses. 

Summary of Ergonomic Data Sources 

Review and analysis of existing Human Factors references were conducted to identify the specific 
proolems of the Human Factors information sources. Problems were identified in three categories: 
awkward representation of Human Factors data, poor Human Factors design application examples, 
and poor Human Factors hands-on examples. 

With respect to data representation in the existing Human Factors data sources, the investigation 
reveals that not all the data were presented in the form most appropriate for use by the designers. The 
excessive use of statistical analysis terminology and technical jargon in data tables and charts was a 
typical example of the awkward data representation category. Human Factors data should be 
presented in the form that most designers can extract valuable information without difficulties. 

Poor application and hands-on examples in the existing Human Factors references were identified as 
problems of Human Factors information sources. Not providing complete explanations for design 
standards in application and hands-on examples caused confusion to the users. Human Factors 
design sources should provide enough application and hands-on examples with clear explanations. 
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Summary of Existing Human Factors Instruction 

Analysis of existing Human Factors instructions was conducted during Autumn 1991 and Spring 
1992 in the Department of Industrial and Systems Engineering and in the Department of Industrial 
Design at O.S.U. Instructional problems which were identified during the investigation could fall 
into three categories: 1) insufficient instruction for the reliability of various Human Factors data 
sources for industrial design students, 2) insufficient instruction for Human Factors data 
interpretation for industrial design students, 3) poor Human Factors application examples within the 
instruction and 4) poor Human Factors hands-on examples within the instruction. 

Insufficient instruction regarding the reliability of various Human Factors data sources was identified 
as a problem of the ID course. Good Human Factors instruction should provide a list of available 
anthropometric sources and a review of the sources. Insufficient instruction regarding data 
interpretation was identified as a problem. Good Human Factors instruction should provide various 
statistical Human Factors data interpretation methods to design students so that they can use Human 
Factors data for their purpose. Finally, poor Human Factors data application examples and hands-on 
examples were identified as problems. Human Factors application examples and hands-on examples 
appropriate to design students should be provided by the instruction. Through these examples, 
students can learn various Human Factors design strategies. 



Problem Statement 

This investigation led to the problem identification. During class projects, industrial design students 
experienced a variety of difficulties as they attempted to use Human Factors data and methods 
during their design development process. Feelings of frustration and hesitation which arose from 
these difficulties caused the students distress in using Human Factors data and methods for their 
design development. 

The existing Human Factors information sources available to industrial design students were 1) 
Human Factors data sources and 2) Human Factors instruction. These two information sources were 
reviewed and analyzed for a clear understanding of the problem. The problem was categorized into 
five groups and they were as follows: 

1. Insufficient instruction regarding the reliability of Human Factors data sources. 

2. Awkward representation of Human Factors data within Human Factors data sources. 

3. Insufficient instruction regarding Human Factors data interpretation. 

4. Poor Human Factors design application examples within Human Factors data sources and 
provided by instruction. 

5. Poor Human Factors hands-on examples within Human Factors data sources and provided 
by instruction. 



Project Objectives 

With these identified problems in mind, project goals and objectives were established. Criteria for 
'good' ergonomic instruction were also devised. The goal of the study is to develop an instructional 
system that can resolve the problems identified in the previous section. The objectives of the study 
are: 

OBJECTIVE 1: To design an instructional system that provides an instruction regarding the 
reliability of Human Factors data sources. 

OBJECTIVE 2: To design an instructional system that improves the quality of the 
representation of Human Factors data sources. 
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OBJECTIVE 3: To design an instructional system that provides an instruction regarding 
Human Factors data interpretation. 

OBJECTIVE 4: To design an instructional system that improves the quality of application 
examples. 

OBJECTIVE 5: To design an instructional system that improves the quality of hands-on 
examples. 

Instructional Goals, Objectives, and Lessons 

Next an instructional system was designed based on project goals, objectives, and criteria. 
TheSu^oL systembuilt up a serifs of goals, objectives and lessons. made 
at this time to develop a computer based courseware program. Industrial design studente^ will 
perSrm^rlous Hum^n Factors design activities. The program was then developed based on these 
instructional goals, objectives, criteria and expert input 

The following section describes examples of an instructional goal and performance objectives for 
Human Srs instruction. The instructional objectives are based on objectives of this project. The 
project objectives led to the instructional goal establishment. 

GOAL 1.0: (Intellectual Skills) 

Industrial design students will identify a list of Human Factors data sources. 

GOAL 2.0: Industrial design students will read Human Factors data correctly. 
Objective 2.1 (Intellectual Skills) ... i • 

Industrial design students will identify the concept of various statistical analysis 

methods. 

Objective 2.2 (Intellectual Skills) , 

Industrial design students will identify various Human Factors terminology 

GOAL 3.0: Lndustrial design students will interpret Human Factors data correctly. 

Objective 3.1 (Intellectual Skills) 

Industrial design students will recognize the characteristics of statistical analysis 
methods used in Human Factors data sources. 

Objective 3.2 (Intellectual Skills) c .„ rc 

Industrial design students will recognize the meanings of vanous Human Factors 

terminology. 

GOAL 4.0: Industrial design students will learn Human Factors design strategies through 

application examples. 
Objective 4.1 (Intellectual Skills) ^»u„ s . 
Given row anthropometric date charts and tables, students will demonstrate their 
understanding of different Human Factors design strategies by answenng a given 
question. 

Objective 4.2 (Intellectual Skills) . . 

Industrial design students will recognize the characteristics of Human Factors des.gn 

strategies through application examples. 
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Objective 4.3 (Cognitive Strategy) 

industrial design students will adopt Human Factors design strategies through 
application examples. 

GOAL 5.0: Industrial design students will experience Human Factors design process 
through hands-on examples. 
Objective 5.1 (Intellectual Skills) 

Industrial design students will demonstrate their understanding of Human Factors 
design process by answering a given question. 
Objective 5.2 (Cognitive Strategy) 

Industrial design students will adopt Human Factors design process through hands- 
on examples. 

GOAL 6.0 (Attitude) 

Industrial design students will be encouraged to choose to use Human Factors basic 
data and design methods. 

GOAL 7.0 (Attitude) 



Content Development 

A major concern of the program development was to develop an instructional system that solves the 
problems identified in chapter I. The problems were: data source access problem, data representation 
problem, data interpretation problem, poor application examples problem, and poor hand-on 
analysis examples problem. The first version of the script had been developed during Spring 1992 
was based on these concerns. In the development process, the instructional goals and objectives 
served as a spring-board to concepts development. 

The courseware was built around the tasks associated with the ergonomic research, analysis and 
problem solving involved with designing a sit-down fork-lift truck. A fork-lift truck was introduced 
in the script development process for: 1) providing good application and hands-on examples, 2) 
drawing design students' attention, and 3) providing a mean of changing student's attitude toward 
Human Factors. First of all, a fork-lift truck design can provide good application and hands-on 
examples because it involves various Human Factors design activities such as access to 
anthropometric data; interpretation of the data; searching for Human Factors design strategies; field 
study process; and design analysis. Secondly, industrial design has been played a major role in the 
development of the equipment and a fork-lift truck design can attract design students because the 
industrial equipment industry is a potentially attractive job market to industrial design students. 
Thirdly, a fork-lift truck design example can change students* attitude toward Human Factors by 
demonstrating that Human Factors can be easily applied to the product development process by 
designers. 

The script was evaluated by Human Factors educators, namely Alan Wier, Joseph Koncelik and 
Mohamad Parnianpour. Based on this evaluation, the program was revised and implemented during 
Spring 1992. The interface guidelines, which were established previously, also played an important 
role in developing the program interface. 

Overview of Content 

An overview of the program content of "Let's Learn Ergonomics." The program is composed of three 
major sections and 14 divisions. 
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The three sections are: 

Section 1 : Initial attitude and intellectual adjustment 
Division 1 : Job opportunity 
Division 2 : Application form 
Division 3 : Statistical terms & Ergonomic terms 
Division 4 : Briefing & Review 

Section 2 : Design analysis and test 

Division 5 : Job description 
Division 6 : Your job plan 
Division 7 : Complaints 
Division 8: Field Study 

Section 3 : Applying Hu?rw , Factors data to product 
Division * : Measuring truck 
Division 1J : Take body data 
Division 11 : Comparison strategy 
Division 12 : How much change 
Division 13 : Z-score 
Division 14 : Recommendations 

In the program, users "apply" for a job with Crown Equipment Corporation, an actual manufa^er 
of SKdling equipment. They then complete the necessary training to become qualified to do 
e gonSc field re Jarch .Af ter competing the field research, the user is presented 
formulas and application strategies commonly used in ergonomic design. Finally, *e user makes 
rec^e^rS for design chLges in the tmck, based on the earlier field research, evaluation 
formulas, and application strategies. 

The first formative evaluation was conducted by Jong-Ho Lee during Spring, 1992, as part of his 
Masters Project. The second formative evaluation was conducted by Robert Jennings, a graduate 
Se in design education, during Autumn 1992. Both evaluations utilized studentsand experts to 
measure effectiveness of the program. Instructional goals and objectives ^ere used as ; cr. ena for 
the evaluation. The evaluation was based on analysis of data gathered from student and expert 
responses to a formal instrument, observation of students using the program, and interviews. 
Conclusions were drawn from this evaluation. Recommendations for further modification were 
derived from those conclusions. 
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BRIEFING 




Standard Doviati«fl« 

Th« sJcndord deviation is a m»a»ur« of th« 
variability of o numbor* around lh« 

m#an. 

For oxompU, 

In o normal distribution, approximately 95% 
©J rh« Ml will bo within j-2S of th« mean. 
(From $ondor> & McCormick) 



in ergonomics, we usuolfy try to 
accommodate 90%of the 
population or almost 2S,0 from 
rhe mean value. 



Figure 1) Division 4 : Standard Deviations 



BRIEFING 




P»»iHmm\ Height 



DoflnltUtt 

The vertical distance from the floor to the 
underside of the thigh imm»diat«Jy behind 
the knee. 

ApplUatUn 

Vertical distance from tho floor to tho top 
forward edge of tho seat pan for the seated 
operator. 



* figure tamM : 

fern o, Jvfcv* 4 Mortm Z***k 

Human Dm — i— o n * Ue»ooc Space 



Figure 2) Division 4 : Popliteal Height 
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MEASURING TRUCK 




D1 : 


Soot cUoronc* 


95 cm 


02 : 


Bock Rttt 


38 cm 


D3 : 


Soot Height 


36 cm 


04: 


Thigh CJooronca 


29 cm 


D5 : 


Irak* DUtono 


38 cm 


06 : 


Hand &rok« DUtanca 


62 cm 


07 : 


Whoal Dutanco 


53 cm 


08 : 


Abdominal Ckoranc* 


43 cm 


09 


Stat Ungth 


41 cm 


010 


: Vision Cloaranca 


48 cm 


on 


: Whoa! Hoight 


39 cm 



Figure 3) Division 9 : Measuring Truck 



TAKE BODY DATA 

Takon from USA* Womon data (5%tiio) 




HI : Sitting Hoight 804 cm 
H2 : Eya Hoight 687 cm 

H3 : Knoo Hotght 45.5 cm 

H4 : Popliteal Haighf 38.0 cm 
H5 : luttock-Knoa Ungth 53.2 cm 
H6 : Thigh CWaronco 104 cm 

H7 : Uppor Arm 30.6 cm 

H8 : Lowar Arm 40.5 cm 

H9 : Elbow Ratt Hotght 1 8.7 cm 
H10 : Abdominal Dtpth \sj an 



* Row* ftourc* : 

Poiwro, Mm 4 Martin UkA 

Human Dimwuion 4 lr*#r ior Space 



^ 

Figure 4) Division 10 : Take Body Data 
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COMPARISON STRATEGIES 



] 



lOOtttU 




Standard 



2L Roach 

in designing obtain foalurtt of our 
built physical world, ono should try 
to occommodato oil (or virtually oil) 
iho popuioHon in question. 
Deigning (or 5h %tiU population 
is 4» appropriate sJrotogy for th« 
com of dUkmc* of a brok« podol 
from i\w operator. If you toko 5th 
%HU fern alt data, you can 
occornmoaate all 1h* poopU who 
oro longer ihon 5i\ %HU bmoU 



Figure 5) Division 1 1 : Comparison Strategies > Reach 



COMPARISON STRATEGIES 



Bo»d on thr M »*ot»gi«, w# <uWgn«d strati.* to tU pom of our fork-lift truck. 




Figure 6) Division 1 1 : Comparison Strategies > Fit 
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Developing Computer-Based Learning 
Courseware in a Large Institution 

WILLIAM SWEETERS 
Director Computer Sen ices 
Independence Community College 
College A\c. & Brooksidc Drive, Independence, KS 67301 

Abstract; A survey of experienced CBL developers in a large institution w as undertaken to 
determine their perceptions of the best and worst uses of CBL development, the perceived blocks to 
development and perceptions of the difficulty in implementing the various phases of the CBL 
courseware development process. Recommendations for managing and developing CBL courseware 
arc included. 

Introduction 

The adv ent of multimedia has brought new and exciting capabilities to computer-based learning. With its 
accessibility and huge storage capacity, CD ROM technology has enabled us to create multimedia programs which 
arc powerful educational tools for use on personal computcrs.Many organizations and institutions will want to 
develop computer- based learning tools enhanced with multimedia. How should they go about this'?. 

This paper presents a study dealing with factors which have effected the development of CBL courseware at The 
Singapore Polytechnic, a large technical institution serving over 13,000 students in ten technical departments. As 
Director of The Centre for Computer Based Learning (CCBL) the author managed a courseware development prxx-css 
w ith a core of five Centre staff mcmbcrs.Thc major source of input to the development of courseware came from the 
600 iccturcrs of the Polytechnic. 

In 1987 the CCBL was established at the Singapore Polytechnic. Since then, the CCBL has promoted the 
development of courseware by holding seminars and conferences and establishing a network for interested staff. The 
Centre has established a development process and offers staff training in courseware design and authoring languages. 
Over 200 staff have attended Centre courses and about 150 courseware projects were attempted over the years. In 
addition, the Centre manages projects by conducting meetings and reviews with staff and setting production 
schedules. The Centre is currently overseeing about 15 courseware projects each year. 

The findings of this survey should be useful to any organization attempting the widespread development ol electronic 
learning courseware, using staff members as the primary developers. 

Methodology 

A survey was sent to 73 staff members who had completed a CBL courseware project within the past two years or 
were supervising an active script. Alter a follow-up notice, valid responses were received from 40 staff while 10 
staff were found to have left the institution. 

In order to validate the survey results by triangulation, an additional means of communication was used to obtain the 
same type of information. Telephone interviews were conducted with a random sample of IS 1 * of the total group. 
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Findings 

The sun cy co\ crcd three ureas of concern: 

What is the best & worst use of CBL courseware? 

What are the blocks to development of CBL courseware? 

How difficult are the phases of the CBL courseware development process? 
Findings are reported by area of concern. 
What is the best & worst use of CBL courseware ? 
Best Use 

The most common reported best use of CBL courseware is as a supplement to the lecture/ tutorial and as a teaching 
aid. 

The nc\t most commonly reported best uses of CBL courseware arc to provide individual student access to material, 
as remedial help, for revision/ study before exams or as a make-up for students who have missed instruction. 

Other best uses of CBL courseware arc to stimulate student interest in a topic, as a prc-lab exercise to help students 
prepare for lab work and to replace paper and pencil exams. 

In addition, a few staff indicated that CBL was best used for - 

♦ teaching 4 the basics' 

♦ teaching theory 

♦ for simulations and 

♦ for practice questions. 

Worst Use 

A majority of the respondents indicated that using CBL courseware to replace the teacher is the worst way to use 
CBL. 

Additional worst uses included using CBL to replace practical lab work and to simply present information, c.«j., 
as an electronic page turner. 

Other worst uses included using CBL to teach psycho-motor skill development and lor drill and practice. 
A few staff replied that a worst use was not to use It at all! 
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WhaT are the J b locks to the development of CBL courseware? 



Figure 1 shows ihc blocks to development. 



Blocks to Development 




■ Too busy 
HI Too long a process 
HI Not a priority now 

Few rewards 
Si Not enough support 
Hi Need team effort 
[J] Too much change 

Other 



Based on frequency 
of response 



Figure 1. SialT reports of blocks to the development of CBL courseware 



Too busy and too long a process 

The major blocks to CBL development are the lack of time available to undertake a courseware project and the 
length of time required to complete a project. These arc not surprising findings since lecturers often have over 22 
student contact hours each week. Assuming only one hour preparation for each contact hour results in a 44 hour 
w ork week. 

In addition, staff have duties on committees, sports programs, counseling and a number of special projects to 
dc\ clop, conduct and manage. There appears to be little time left to undertake a CBL dc\ clopnicnt project. 

In addition to the lecturers lack of time for undertaking projects is the great length of lime needed to dc\clop CBL 
courseware- 250 hours to develop a 1 hour lesson is common. It typically takes 6 months to a year to produce a 
lesson. Nc\ crthclcss, Polytechnic staff have been producing about 15 courseware per year. 

Not a priority now and too few rewards 

Some staff reported that CBL dc\ clopmcnt was considered a low priority compared to their other duties. Stal l also 
indicated that there were few rewards given for producing courseware. 

Not enough support 

Stuff mentioned a need for facilities to support CBL development. Staff need ready access to computers in their 
offices and homes. 

Computer loans and special staff discounts can be negotiated when institutions place large orders for computers. 

Students also need ready access to view computer-based courseware. Booking terminals in computer labs is often 
troublesome to students - computers and software need to be a\ailablc campus wide. Resource centers in 
libraricsjounge areas and dormitories can provide the required student access to computers. 
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CBL development requires a team approach 

Staff indicate thai a team approach will ensure thai topics chosen for CBL dc\ clopmcnt will be integrated into the 
curriculum. Staff also mentioned that some curriculum management committees arc not even aware of CBL 
capabilities. 

In a particular course, one hour of CBL is not likely to have much impact but 10-20 hours of CBL can pro\ ide 
enough variety and focused skill development to significantly improve student motivation, interest and 
achievement. A team approach can provide a more focused plan for CBL development and bv getting more people 
invoh cd in development, more CBL courseware can be produced for greater impact. 

Syllabus changes too fast 

Some staff dc\ eloped courseware on topics that were later dropped from the curriculum. Others ha\ c developed 
courseware lor their courses and then been reassigned to different courses. Such experiences have discouraged staff 
from developing CBL. 

II CBL projects were planned by a team effort and coordinated through curriculum management committees then 
topics likely to be dropped or modified could be identified and rejected as candidates for CBL development 



How Difficult are the Phases of the CBL Development Process? 

Figure 2 shows how staff rated the difficulty of each pha.se of the development process. 



Level of Difficulty [ 



4- 




Choosing Analysis Scripting Coding Evaluat 



ion 



Figure 2. Level of difficulty for each phase of the CBL development process 

As might be expected, coding courseware is perceived to be the most difficult phase of development by the staff The 
other phases of development fall in mid-range and arc viewed as neither particularly difficult or easy. Scripting is 
rated as the second mast difficult activity and no phase is considered easy. 

Indeed, coding requires programming skills which arc often quite alien to lecturers. In the normal course of prcpanng 
lor classes the skills o! choosing topics, analyzing content, writing lecture notes and evaluation arc utilized but 
programming skills arc not required. Coding is a bottleneck in the development process and must be addressed The 
Centre oilers stall training in authoring languages and help for special coding problems but this is not enough 
•support lor most staff. The Centre has taken responsibility for much of the task of coding and codes scripted lessons 
by assigning projects to Centre staff, hiring programmers, hiring external consultants to ecxlc and using students to 
code lessons as project work m their field. Students have been espcuailv keen on coding courseware as thev enjoy 
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working with an authoring language and developing computer graphics. Stall' report that because ol our support in 
taking over much of the burden of coding, they arc more motivated to embark on CBL development. 



Many other blocks to CBL development may not have surfaced in this study because they ha\ c been anticipated and 
addressed by Centre staff. The Centre offers courses in CBL design and authoring languages, models for development, 
screen design guidelines and regular reviews with staff. These activities support the development process and there 
was no criticism of Centre procedures. Were any of these elements missing, they might surface as additional blocks 
to development. 

Conclusion: 

Best Use of CBL Courseware 

Staff suggest that the aim of CBL courseware should be to support rather than replace traditional instruction. CBL 
courseware can be used to deal with many of the problems associated with lecturing, e.g., remedial help for students 
unable to make satisfactory progress, revision/ study aid and as a make-up for students who miss instruction. 

Worst Use of CBL Courseware 

Staff arc concerned that CBL courseware could be used to undermine ; ic importance of the lecturer/ tutor and to 
replace direct teaching and laboratory experience. 

Blocks to Developmetr 

Many blocks to develops * ^tcm from the organizational goals of the institution. If electronic learning is promoted 
from the lop down, it will become a priority goal. Closely related to priority goals arc the rewards offered to staff 
who develop CBL courseware. Stal l who undertake a CBL project could be allocated specific time for it in place of 
scheduled classes, they could be given the kind of recognition a publication brings towards merit awards or tenure, 
they could be given direct monetary awards for courseware used by the department and they can be given personal 
recognition by the department or institution. Some staff mentioned that their department heads and colleagues were 
not aware of the time and effort required to produce CBL courseware. 

The major blocks to CBL development arc the lack of time for stal l to undertake a project and lack of an 
organizational commitment to allocate staff and resources to the goals of CBL courseware dc\clopmcni. 

Phases of Development 

Coding is the mast difficult phase of CBL development for staff and all phases arc moderately di fficuit. 

Recommendations 

The following recommendations should be considered by any organization embarked on the widespread development 
of CBL courseware: using staff as a primary resource: 

'A Provide independent coding support 

A major block to CBL development is the effort required to code courseware. It is necessary to break the coding 
bottleneck by providing coding assistance to staff developing CBL courseware. 

■flr Provide top down support and rewards 

An institutional commitment needs to be made to support CBL development in an organization. A rewards system 
for staff efforts needs to be established. 

Jr Integrate CBL development with curricula 

CBL development should not be let t to the initiative of individuals. Courseware development needs to be planned 
with the curriculum management committee and a team effort needs to be started to ensure that a reasonable number 
ol courseware will be developed. 
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Establish a CBL development unit 

CBL development within an organization needs the control and guidance p»v>\ ided by a development process, design 
guidelines, models and templates. Developers also need support such as training, re\ icv\s of their work, seminars and 
networks to promote ideas and motivation. They also need a representative to express their needs and concerns to 
departments and the organization. 



Suggested Readings 

Bun/el, M.J.,& Morris, S.K. (1992). Multimedia applications developments: Usini> DVI Technology New York- 
McGraw-Hill. ' 

Murphy, S.J. (1990). A manager's guide to audio-visual production. London: Kogan Page. 

Sammons, M.C. (Feb. 1994) Motivating faculty to use multimedia as a lecture tool. T.H.E. Journal, 88-90. 
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Exploring the Reasons Behind Design Decisions: 
Interactive Multimedia Learning Environments 

WARD MITCHELL CATES, ED.D. 
Leadership, Instruction, and Technology, Lehigh University 
111 Research Drive, Mountaintop, Bethlehem, PA 180154793 USA 

E-mail: WMC0@LEHIGH.EDU 



Abstract: This paper focuses on decision making in the instructional design process. In particular, 
it explores the key decisions made in designing an interactive multimedia learning environment 
(IMLE) to teach science to bilingual students in 7th to 1 1th grades. Key decisions include handling 
content area instructional demands, selecting metaphors and images, reducing cognitive load, 
supporting learners through help, advisement, and coaching, and selecting appropriate features and 
capabilities. Attaining a balance between direct instruction and learner exploration and 
experimentation is also addressed. 

As Mountford (1990) noted, "There are few success stories for interface designers to learn from" in the field of 
instructional design (p. 18). In the absence of success stories, she suggested that designers share the results of 
their efforts with one another. There is no formula for *good* design. Design is a creative process based on 
carefully reasoned decisions. Too seldom, however, do designers discuss the reasons behind their decsions. This 
paper discusses the design of an interactive multimedia learning environment (IMLE) for teaching science to 
seventh through eleventh graders in learning situations where students may have as their first language either 
English or Spanish. I have used the Jason Project as the basis for a proof-of-concept demonstration, illustrating 
how some of the concepts in the design might be implemented using material from that project, and this paper 
uses examples from that proof-of-concept treatment 1 . The paper is organized around a scries of questions, the 
answers to which explain why the product under design looks and operates as it does. 

IMITATE OTHERS OR BUILD ANEW? 

When asked to design a new product, one has two choices: One can imitate some previously designed piece of 
software or one can create a new design from scratch. When designers have done careful and creative design, 
others can benefit from adopting their decisions. Similarly, if users have become accustomed to those designs, 
they should have less difficulty adjusting to products employing similar designs. Like many products on the 
market, the current product chooses to employ some common elements of other designs, including the use of a 
control strip of icons which makes available to users a variety of functions (Iuppa & Anderson, 1988), electronic 
journal capabilities (Civil War Hypermedia Project Design Document, 1991), on-line context-sensitive help 
(Scllcn & Nicol, 1990), and the ability to leave the program and restart at the same point (Brown, 1988). The 
control strip is familiar to many users and seems likely to help users think of the functions on the strip as a 
visual entity. That is, the visual unity of the strip may help users perceive it as a whole, thus helping to group 
it as a single * chunk,' reducing the cognitive load on the user (Miller, 1956). The electronic journal permits users 
to record thoughts as they occur to them and to make notes as they deem necessary. This, too, should increase 
the sense of learner control and reduce the cognitive load that memorization or attempting to perform actions 
while recalling one's own thoughts may impose (Orcn, 1990). On-line context-sensitive help should help cure 
the problems of user disorientation and, once again, reduce the cognitive load imposed by having to recall where 
one is and how one got there (Baird, 1988). The ability to leave and restart should make learners feel more in 
control and should help make the product better suited to the demands of instruction in settings where time may 
be limited (Gates, 1992). 
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HOW SHOULD LEARNER NEEDS BE SUPPORTED? 

Hutchings. Hall. Briggs. Hammon. Kibby. McKnight, and Riley (1992) argued that "raw hypermedia is not sufficient 
to support effective learning. Educational hypermedia must be able to support a variety of interaction/learning 
stytes (p. 177). Jonassen and Grabinger (1990) and Nicol (1990) concurred, suggesting that hypermedia designed 
to facilitate learning outcomes needs to address learners' information processing and the demands of the tasks they 
are trying to accomplish. Spiro and Jchng (1990) argued that, "one learns by criss-crossing conceptual landscapes- 
instruction involves the provision of learning materials that channel multidimensional landscape explorations under 
the active initiative of the learner" (p.170). Gygi (1990) expressed reservations about broad large information 
vistas, however, suggesting that they can be daunting. Many writers have addressed the cognitive load that large 
dataspaccs' place on learners (sec for example. Lanza & Rosclli. 1991 and Marchionini. 1988). Some, like Mitsch 
and Dubbcrly (1990) and Scllcn and Nicol (1990). have suggested that conceptual maps might help learners navigate 
such spaces. The findings on the effectiveness of such maps have not been positive, however; it appears that they 
may be too complex to assist learners in any significant way (Jonassen & Wang. 1993). A second approach might 
be to reduce the size of the learner's universe'; that is. to focus more on depth of coverage than on breadth of 
coverage and to select for inclusion only pieces of information likely to assist learners with the material at hand 
This approach has received the support of a number of writers (sec for example. Florin. 1990; National Committee 
on Science Education Standards and Assessment, 1993; Padilla. 1990 and Sivitcr & Brown 1992). It is the 
approach used in the present product. 

As noted in the previous section, one way to support learners is to make good on-line help available from every 
screen. This help is not intended to replace good interface design, but rather to eliminate the sorts of confusions 
that learners, particularly novice learners, experience when first working with a system (Carlson & Grabowski 1992- 
Rhcmgold. 1990). Scllcn and Nicol (1990) suggested that such help might supply the answers to five tyWs of 
qucsuons: What can I do here? What is this and what does it do? How do I do this? Where am P What just 
happened and what docs it mean? On-line help in the present product is designed to answer these sorts of qucsuons. 
This type of help is normally referred to as 'context-sensitive' help. 

Another type of help might be called 'content-sensitive' help. This would be advice or coaching based on the 
material available from a particular screen. Marchionini (1989). Wolz. McKcown. and Kaiser (1989). and Yordy 
(1^91). among others, have suggested that such advice may help support learners' needs. In the present product 
content-sensitive help includes advice on what to look for. advice on what scientific concepts are involved and 
advice tn the form of Socratic questions designed to direct the learner's thinking (Kccgan. 1993; Steinberg 1992) 
AH content-sensitive help is presented using the language of self-attribution ("I would like. . .") to encourage'a sense 
of learner control (Clariana. 1993; Keller. 1979). Laurel (1990) suggested that person-like guides were popular with 
users. As Brcnnan (1990) noted, people seem to bring expectations from their human to human interaction to their 
interactions with computers. In fact. Oren. Salomon. Kreiunan. and Don (1990) found that users attributed 
emotional characteristics to their guides and formed 'relationships' with them. In order to take advantage of this 
occurrence, the present product allows learners to select their guides from among a large number available, and 
allows learners to interview' perspective guides/advisors before selecting them, by examining their resumes and 
watching a three minute video clip in which the advisor tells a little about himself or herself. Once an advisor is 
selected, that advisor's picture appears on every screen on an icon offering content-sensitive help and advice. 

WHAT TYPES OF ACTIVITIES SHOULD BE INCLUDED? 

There appear to be four main ways in which a multimedia product might be employed in instruction: (1) to deliver 
TZrlTTwT'^! 0 ' f*Pj ora ^J eamin g. 0) as a research tool, and (4) to prepare multimedia presentations 
(Grccno 1991). Each calls for a different level of learner control. There is a wide body of research on learner 

mT 1C 0 aniCrS have a grcat dcal of difficull y makin 8 choic <* "nd handling their own learning needs 

Heller. 1990; Lanza & Rosclli. 1991; Ross & Morrison. 1988; Santaigo & Okey. 1992). As Jonassen and Sg 

S^!^^ 10 ^ thr ° U8h 3 kn ° WlCd8C bUSC d0CS nW cn « Cndcr CIl0U * h P roccssi »8 to result in 

Direct instruction in IMLEs is often delivered using 'guided tours.' Guided tours resemble traditional classroom 
pre.senUU.ons ,n that they exhibit a high degree of linear structure and they usually have fairly well define! ielin" 
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objectives These properties of guided tours tend to associate them more closely with program control i tiian 
Smer control " alih^lcamm still have some opUons in most guided tours. While exploratory learning otters 
^o^Xteta«» to have difficulty, it also gives learners the freedom to ^criss-cross infonnauon 
landscapes,- examine materials of interest, and explore in ways that invite the learner to form serendip ; tous and 
fortuitous new connections among pieces of information. When a product is used as a research to* * penmb 
Mch» or students to seek answers to their questions using the resources contained in the product Once again 
Smers who are not experienced in such multimedia research efforts may find themselves ^**™» 
learners use a multimedia product to prepare a presentation, they are using Us ^ * <**>*^ » 
use the resources and materials of the product to build a presentation to show o the rs Th use wears to be 
satisfying to many teachers and students and has received much praise in the literature (Collins. 1991, 
McCormack, 1992). 

The present product uses a combination of these activities. Expeditions provide the general structure; each is, 
Sto^nSSdSir of a realistic experience, a form of anchored instruction (The Cognition and Technology 
Group at Vandcrbilt, 1993). At the same time, each offers the learner chances to participate in simulations and 
to conduct experiments, in keeping with American Association for the Advancement of Sciences emphasis on 
^^^LL^^ctaJi^ below). To help learners avoid getting lost in these activities, the adviso 
i, a ways available to keep them on-task and on-targct Learners who don't want or need adv.ee can choose not 
o call it up The product employs a non-intrusive coach; if learners do not click on the advisor icon the only 
%Z *£ will see is that offered to all learners in on-screen instructions and in general adv.ee troir > Bob 
BaS (Son Project host). The product should also offer learners the opportunity to do some unstructured 
expSng on UieU own and, like most products on the market, should oiler at least minimal producuon toddies 
so that teachers and students can produce their own presentations and reports. 

WHICH METAPHORS ARE APPROPRIATE? 

Manv authors have argued that using metaphors in multimedia products can help users, particularly novice users. 
Sst^Xw a P Luct works L predict correctly how to accomplish ^cific acuons (seej or example 
AHinson & Hammond. 1989; Erickson. 1990; Mitsch & Dubberly. 1990; and Roscndahl-Krciuimn, 1990 In 
The example of a product based on the Jason Project, the underlying metaphor for most screens is the expedition 
rtZSm^ by printed materials with illustrations. Where experiments are to be conducted the underlying 
mSphor of a laboratory is employed, with its attendant tools and devices - auxiliary metaphors (see Cates 
So Where simulation or exploration are to employed, the underlying metaphor is that of "real space." Tha 

s i. the ex P oration or simulation is to take place under water, the learner should sec what the « jameras reveal 
to a person manipulating the actual underwater vehicle. If the simulation or explorauon is to take place on the 

urtoeTrZrorout of the water, the learner should sec whatever he or she would be able to sec r«m 
?hc Sace^example, from a small boat following a migrating whale) or standing on the site (for example, 
in a rookery on the Galapagos Islands). 

in terms of iconic metaphors, since the Jason Project has emphasized science on or under or the water in its 
exSons me images used for the icons echo maritime themes (Cates. 1993). For example, navigation be ween 
screens is accomplished using icons displaying images matched to how one navigates on a particular expedition: 
n T Grea Lttkes. the icont employ frigate ships facing forward or backwards. In the Galapagos, he icons 
Imploy Sdivers swimming to the left or to the right Off Baja California Sur the icons ^ploy d« ^son 
underwater vehicle with its manipulative arm extended, cither facing forward or backward. The image for quit 
is a skindivcr returning to the surface. Since Bob Ballard serves as the host for the expeditions, his image 
appears on the icon learners use to obtain general explanations of how to use the product In keeping with the 
product's decision to employ individual advisors selected by the learner, the image on the icon for content- 
sensitive advice is that of the advisor (coach, mentor) selected by the learner. In addition, the remaining icon 
on the control strip employs an image appropriate to the subject area material to be discussed there. Thus, when 
geology information is available, this icon displays an image of the earth; when informauon about broadcast 
technology is available, the image displayed is a satellite dish; when the information available is about how to 
pack for a trip, the image displayed is a suitcase. Baird and Pcrcival (1989) and Gygi (1990) expressed concern 
that users may misinterpret the function of an icon, particularly if they arc uncertain why it displays the image 
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^m d >nnc I ?, k T in i W , ith 1,10 su S8 cslions of Iu PPa and Anderson (1988), Rubin and Milncr (1988), and Florin 
(1990), icons display both images and words to help make clear their functions. 

HOW SHOULD SCIENCE BE TAUGHT? 

White Fontana, and Catcs (1991) argued that the design of an IMLE should be driven by the instructional 

deSlm ST "7 'r *? lCChn ° l0gy - ln kCCping With lh3t argUmCnt > this P^ 1 «** of its 

tT 1°, thedcmands of science instrucuon. In Science for All Americans (1989), the American Association 
for the Advancement ot Science (AAAS) suggested that science instruction be changed to "concentrate on the 
quality of understanding rather than the quantity of information presented" (p. 145). Science instruction would 

h ,h °h \ m0dC] °T limkCd brCadth of what was covcrcd in oroer 10 allow more time to deal with it 
n depth and that viewed details as ways of enhancing student understanding of what they were studying not as 
the basis for memorization and testing. The AAAS plan suggested that, "concepts arc learned best when they 
arc encountered in a variety of contexts and expressed in a variety of ways, for that ensures that there are more 
opportunities for them to become imbedded in a student's knowledge system" and argued that, "students cannot 
learn to think critically, analyze information, communicate scientific ideas, make logical arguments, work as part 
of a team, and acquire other desirable skills unless they are permitted and encouraged to do tnosc things over 
and over in many contexts" (pp. 145-146). 

AAAS offered eight strategics for transforming science instruction (p. 148): (1) Start with questions about 
nature. Work from the lam.l.ar. (2) Engage students actively. Let them manipulate and explore. (3) Concentrate 
on the collection and use of evidence. Students need to be able to separate the relevant from irrelevant 
Students need guidance encouragement, and practice in collecting, sorting, and analyzing evidence, and in 

S^SfETS ^ ° n I"'" ?hCSC aCUviliCS nCCd 10 lead 10 StimuIalin S intcrcst W conclusions for the 
student. (4) Provide historical perspectives. Students need to learn about tiic twists and turns in investigation 
and theory, about the role individuals play in scientific advancement, about "the interplay between evidence and 
Uicory over ume . History is important for the effective teaching of science, mathematics, and technology also 
because ,t can lead to social perspectives - the influence of society on the development of science and 
technology, and the impact of science and technology on society." (5) Insist on clear expression Poor 
communicauons skills can confound inquiry and reporting. (6) Use a team approach: "Scientists and engineers 

ShZ V" gr ° UPS 1CSS ° fte " aS iS ° latCd invcslj gators. ... In the context of team responsibly 
leedback and communication become more realistic and of a character very different from Z u uS 
indrndualisnc tcxtbook-homcwork-recitation approach." (7) Do not separate knowing from f nSg out "In 
science, conclusions and the methods that lead to them are tightly coupled. The nature of inquiry depends on 
what is besng investigated, and what is learned depends on the methods used." (8) De-emphasis the 
memorization of technical vocabulary. Introduce technical terms only as needed and gradually h* StaWM 
up a vocabulary of used and demonstrated terms rat... than having them memorize Isolated Lings Z tor™ 

The product attempts to meet tiic spirit of AAAS' goals and to implement the strategics presented above The 
design strategics for accomplishing this are discussed in the next four subsections. 

Addressing the Wa y Scientists Think and Work 

Instead of memorization the product uses frequent manipulation and experimentation with repeated exposure to 
Pn,S 7 , and c malCna S - C A glossaf y of scicncc lc ™ " always available and definitions -ire stiucdTpTa n 
Si >• Z la, " Spamsh) ' Sludcnts d0 " ol havc 10 KO through lots of direct instruction before moving on to 
manipulafon and expenmcntauon. Learner creativity and curiosity are encouraged, as is a practical orient 
For example, upon selecting an expedition on which the learner wishes to go. he v J^SlS^lS^^ 
h.s bag paymg attention to the demands of the region and the weight restrictions of u- Je Slhls am 
portrayed as hard working, persistent, creative, and practical. Learners are exposed o^vc^SutioVw 
scent, ,c questions and anecdotes and stories that illustrate scientific creativity P Example? u t^t now n 
undcrstandmg of science can help us explain, understand, and predict phenomena, and l^t?-! 
practical problems and explorations on each expedition that encourage tnem to deduct Xo^ ciS 
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The product emphasizes the fact that scientists are members of a larger community, and that this community, 
through peer evaluation and cooperation, protects against unwarranted assumption or interpretation. The product 
makes it clear that there is much disagreement in the scientific community, and that disagreement is viewed as 
a healthy part of the peer evaluation process. The product points out that many now-well-acccptcd concepts 
initially had much difficulty gaining acceptance and helps to show that debate helped to facilitate acceptance. 
The product highlights controversy and disagreement, emphasizes point-of-vicw, and illustrates vested interest. 
In some exploratory activities, students arc asked to assume roles or to work on scenarios that help them 
experience the drive to seek "desirable" results or findings. The product illustrates through examples the 
changing acceptance of scientific theories and concepts. Students are asked to consider and explore ethical 
issues in science, including the need for and use of intrusive practices (like dissection). To help students 
recognize the role of models and systems, the product uses technological tools as examples of systems with sub- 
systems. It also uses ecological systems and sub-systems to make the point. The product demonstrates how 
rr'vicls can be used to explain occurrences, and asks students to use models to make predictions. The 
relationship between change and stability is used to help students recognize the larger patterns in events. 

Making Instruction Match the Demands of Science 

The product asks the student to form and test hypotheses, collect and analyze data, and draw conclusions. Its 
advisors attempt to provide the same sort of intellectual and instructional support that good teachers of science 
provide. The product should supply enough activities to meet or exceed the typical curricular emphasis placed 
on this type of material, thus permitting teachers and students to use as much, or as little, as their schedules 
permit. Some of the product's activities arc individual learner activities, some arc small-group activities, and 
some are computer-assisted whole-class activities. To foster a greater sense of collaboration and peer evaluation, 
the product uses simulated "e-mail" among teams, to and from individual students, and between the teacher and 
students or teams. The product provides screens in which learners arc required to 'construct' tools from parts 
available and then to manipulate these tools to collect and analyze data. The product provides simulated 'real- 
world' experiments, accompanied by appropriate comments and video segments involving scientists. Students 
arc encouraged to keep a journal, to update it often, to reflect on the currency of previous entries, and to note 
questions as they occur. Advisors stimulate learners to make entries in their journals and to propose both 
hypotheses and conclusions. 

Helping Students Understand Science's Role in Society 

Where appropriate, the product explores how societal opinions about the relative worth of different people, 
events, ideas, and objects in our society change. It notes that all peoples have both noble and ignoble records 
of societal development in their evolutions, and illustrates how opinions have changed and addresses specifically 
how society's change contributed to or impeded change in scientific opinion. The product makes s that there 
arc conflicting, strongly-held, and often logical opposing opinions on many scientific issues. Students arc 
encouraged to form their own opinions, to articulate them clearly, and to defend them in the face of opposition. 
Once learners have formed their opinions, the product may introduce dissonant information to challenge those 
opinions. Students arc guided to ?.oc that, in real life, there are few easy answers. 

Addressing Technology's Role in Science 

The product explores how improved technology can produce improved data and how those, in turn, can produce 
improved findings and interpretations. Simulations and manipulations help students sec the value of new 
technologies themselves. Manipulations, simulations, and explorations often call for an engineer and a student 
serve. this capacity, learning in a practical setting about the interaction between engineering and science. 
Through direct instruction and explorations, the product examines some of the side effects of technological 
development. Some of the events in which students participate experience technological failures, sometimes of 
a sub-system to reinforce the concept of sub-systems. The product offers both complex technological solutions 
and simple technological solutions with accompanying failure rates in order to help students appreciate the 
influence of Murphy's Law. In some simulations, students find themselves faced with decisions precipitated by 
technological advancements. They arc asked to realize the difference between "technology permits us to do this" 
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and "it is the right thing to do" (Bacz, 1991). For example, should we remove bodies from a sunken ship? 
Should we eradicate the mosquito? 

HOW CAN THE PRODUCT ADDRESS DIVERSITY? 

Much science instruction does not appear to expose girls to strong female role models in science education, role 
models that may help girls envision themselves as scientists (Hudlcy, 1992). In some science classes girls appear 
to be treated differently than their more aggressive male classmates and may end up as watchers rather than doers 
(Potter & Rosscr, 1992). In classes where memorization of specialized vocabulary constitutes much of the 
instruction, Spanish-speaking ESL students may find themselves similarly allocated L> a secondary position 
(Warren, Roscbcry, & Conant, 1989) Girls may approach things more collaboratively L'uin boys and teachers 
may be rewarding boys for competitive behaviors not necessarily reflective of practice in real-world science 
settings (Kay, 1988). 

In order to play on the need for positive role models, the product allows learners to select advisors/guides 
according to their own personal preferences. Some advisors/guides arc female, some are African- American, and 
some arc members of Spanish-speaking cultural groups. All science advisors/guides should be charismatic and 
intellectually attractive. In all video segments and scenarios, female, African-American, and Spanish-speaking 
students arc portrayed as "doers" of science, not "watchers" or "assistants." The product encourages both 
cooperation and competition in its activities. This should not only help students learn that most scientific 
accomplishments occur because of collaboration, but may help female students see a new role for themselves 
in science (Adams & Hamm, 1990; Stephenson, 1992). 

The product is bilingual (Spanish-English, English-Spanish) in its delivery. The product docs not relegate 
Spanish to the "translated version for 'English-impaired' students," but instead portrays Spanish-speaking scientists 
as capable and important members of the world scientific community. This should help Spanish-speaking 
students see themselves as having a proper place in that community. The product can convert all text to Spanish 
if so desired, including both screen text and audio soundtracks. The product presents all learners with some 
bilingual skills, teaching Spanish vocabulary as appropriate in order to help all learners understand the Spanish 
place names (for example, Las Tortugas), descriptive labels (for example, punta, isla, laguna), and milking 
presentations for Spanish-speaking people (for example, the dedication in Spanish by the Ecuadorian officer on 
the repatriation of the iguanas) that are associated with the scientific explorations. As non-Spanish-speaking 
students work with this product, they should gain useful vocabulary and a sense that knowing such vocabulary 
expands their knowledge and helps them to become members of the world's scientific community. As Spanish- 
speaking students work with the product, they should sec themselves at a slight linguistic advantage (for a 
change) and should feel some sense of pride in their bi-cultural membership (DeVillar & Faltis, 1991; Schcel 
& Branch, 1993). This pride may, in turn, enhance their pride in Spanish-speaking members of the scientific 
community as well as help them to recall that (I) much of the world speaks Spanish, (2) many places were 
discovered, explored, and named by Spanish-speakers, and (3) the practice of science is as active in Spanish- 
speaking countries as it is in the rest o[ me world (Bcntlcy & Ditchficld, 1989). Similarly, where possible, the 
product utilizes appropriate examples of African and African-American contributions to science (Bailey, 1990). 

WHAT NEXT? 

As noted earlier, the product described in this paper is in the design stage. As this paper demonstrates, the 
process of design requires much careful thought, and additional work with bilingual instructors and subject-matter 
experts should help to enhance the final product. For an instructional product to make it to market, it must have 
an audience, there must be an instructional need, it must be well designed to meet that need, it must demonstrate 
to its potential users that it is a useful product, and it must acquire the necessary funding. The next step for this 
design is to create a small prototype and test the design on users. The journey from design to market is not free 
of its own set of perils. With careful planning, hard work, and a little bit of luck, this design may survive die 
trip. 
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Abstract: Researchers are developing automated instructional design systems which guide 
subject matter experts (SMEs) through the complexities of courseware development (Spector, 
1993). Enabling SMEs to perform many of the authoring activities associated with courseware 
development has two distinct advantages: (1) costs are contained, and (2) SMEs can make 
optimal use of new technologies. This paper will focus on a research effort called the Guided 
Approach to Instructional Design Advising (GAIDA). GAIDA incorporates case-based 
reasoning in a hypermedia framework in order to convey instructional design expertise to 
novice courseware developers. The paper reviews the theory and development of GAIDA, 
including interim results of an extensive evaluation effort. 

As computer-based instruction (CBI, to be interpreted broadly) has matured, automated environments 
to support instructional design and development have become commercially available. At the same time, 
computer-based technologies have become richer and more complex with the advent of such advanced 
interactive technologies as digital audio/video, interactive simulations, and virtual worlds (Spector, 1990). 

As a consequence, an organization which has the task of producing CBI must confront two difficult 
issues: (1) where to obtain the expertise required to produce quality CBI; and (2) how to make it cost- 
effective to produce a variety of CBI solutions to satisfy various training requirements. There is an industry- 
wide shortage of instructional technology expertise, which makes a near-term solution to the first issue 
difficult. Automated tools are expected to provide some support for the second issue. The system described 
in this paper, Guided Approach to Instructional Design Advising (GAIDA) represents an attempt to respond to 
both problems (Merrill, Li, & Jones, 1990). 



An attempt to capture instructional design expertise in an automated system must address these two 
fundamental issues: (1) What distinguishes novices from experts in this particular domain? and (2) What is 
the purpose of the design advisor? With regard to expert-novice differences, the expectation based on findings 
in cognitive science is that experts perform complex procedures automatically, without consciously attending 
to specific steps (Anderson, 1987; Duchastel, 1990). Novices, on the other hand, are inclined to perform 
complex procedures one step at a time, without looking ahead or considering the overall problem setting. The 
first issue, then, is to determine if this generalization about novice-expert differences holds true in the domain 
of instructional design. An extensive study of expert and novice instructional designers indicates that this 
generalization is indeed true (Rowland, 1992). Moreover, experts tend to proceed from the problem statement 
(e.g., design instruction to support a particular set of instructional objectives with regard to some content area) 
to a framework which is likely to be appropriate for the task at hand based on past experience (Rowland, 
1 992). In other words, experts begin with a case construct that is known to be effective and they then modify 
that case-based framework to accommodate the particular details of the new task. 

Novices, on the other hand, are inclined to follow procedures which are codified in sets of instructions 
commonly referred to as instructional systems development (1SD). The Dep. nent of Defense (DoD) is 
probably the largest single user of ISD. The Air Force has recently devoted si 0 .iificant resources to updating 
these procedures to reflect current academic thinking about instructional design (Tennyson, 1993). ISD is 
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intended to provide instructional design guidelines, among other things. The intent is not to prescribe specific 
steps to be followed inflexibly no matter what the situation. However, these guidelines are placed in the hands 
of novice instructional designers who have no other recourse other than following the specific steps specified 
in the various ISD procedures. The results of instruction designed in these circumstances is not always 
optimal. When the instruction is computer-based, this process can result in expensive but ineffective 
courseware. Because the DoD is facing financial and personnel cutbacks, the shortage of instructional design 
expertise is expected to worsen while requirements for high quality courseware increase. 

In response to this situation, Armstrong Laboratory has been studying various means of enhancing the 
performance of novice courseware developers. The basic premise is that the Air Force cannot afford to 
contract out the development of the courseware needed to maintain operational proficiency and readiness. 
Therefore, performance support tools must be provided to enable novice but motivated courseware developers 
to perform as if they had years of experience. Gagn6 proposed a relatively simple solution to this problem: 
Provide novices with easily understood high level guidance along with completely worked examples and they 
will perform as if they were advanced apprentices (1991). Gagn£ further proposed that the appropriate high 
level guidance could be constructed around the nine events of instruction (1985). The completely worked 
examples should reflect frequently encountered instructional objectives, such as identifying a part of a device 
or carrying out a test procedure on a device. Figure i is a screen from GA1DA which illustrates the nine 
events of instruction. Figure 2 shows the instructional objectives currently supported in GAIDA. 











What is GAIDA? 







GAIDA - Guided Approach to 
Instructional Design Advising 

GAIDA is a PC program providing 
guidance for creating effective 
computer-based instruction (C3I). 
The program is based on Professor 
Robert M. Qagne's ideas fJJ . 
GAIDA guides a novice instructional 
designer through the steps needed 
to produce quality CBI fJ?J, 

The guidance consists of Gagne's 
nine events of instruction. The 
events of the lesson occur roughly 
in order as follows, although the 
order is not considered inviolable: 



The Nine Events of Instruction 



1.. Gain attention. 

2. Describe the goal. 

3. Stimulate recall of prior 
knowledge. 

4. Present the material to be 
learned. 

5. Provide guidance for 
learning. 

6. Elicit performance. 

7. Provide informative 
feedback. 

8. Assess performance. 

9. Enhance retention and 
transfer. 



v 



Figure 1. GAIDA's nine events of instruction. 
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What is GAIDA (continued) 



Do not misinterpret the nine events of 
instruction as a rigid step-by- step learning 
mode). The purpose is to help you plan 
lessons. These nine elements are 
important components of any lesson, and 
frequently proceed in this general order. 
However, they probably don't get equal 
time. For example, presenting the material 
or providing learning guidance is usually a 
much larger chunk of a lesson than is 
stating the goal or recalling prior knowledge. 

In this program, we will present you with 
examples of lessons that have been created 
with Gagne's nine events in mind. We will 
identify where those events occur in the 
lesson. 



■ Guid 
for si 
learr 



Guidance and a sample lesson are provided 
for several learning objectives. The 
learning objectives are liste d below: 



Learning Objectives 



Identification 
Classification 
Procedure (with checklist) 
Procedure (from memory) 



Go to the next page to select a learning 
objective. 



Figure 2. GAIDA instructional objectives. 



In summary, the theoretical framework for GAIDA is drawn from two sources. From cognitive 
science, GAIDA proceeds on the assumption that providing novices with on-line examples or cases will make 
up a significant portion of the difference between novice and expert instructional design performance. From 
instructional science, GAIDA proceeds on the assumption that elaborating the nine events of instruction will 
provide an easily understood and meaningful framework for a novice designer (for additional elaboration of 
integrating theoretical frameworks, see Gagn6 & Merrill, 1990; see also Spector, Poison, & Muraida, 1993). 

Implementation of an Instructional Design Advisor 

Armstrong Laboratory decided to test Gagnd's hypothesis. Gagne agreed to come to the laboratory as 
a National Research Council Senior Fellow and design the system that he had proposed. Gagne argued that 
GAIDA should first attempt to support a difficult design task, such as training students to memorize a 
complicated procedure. The School of Aerospace Medicine at Brooks Air Force Base provided such a case: a 
procedure to test a patient's respiratory capacity. Students have to learn how to set up a device called a 
spirometer, how to connect a patient to the device, what to tell the patient to do, and how to interpret the graph 
produced by the machine. The sample computer-based lesson and the accompanying design guidance were 
created under the close supervision of Gagn6 and a subject matter expert (SME). The original system was 
coded in Asymetrix 1 ToolBook (a high level, object-oriented language for the Microsoft Windows 
environment on 80386/80486 personal computers). Because GAIDA was itself a piece of instruction, Gagne 4 
wanted to include an opportunity for users (novice developers) to practice applying the guidance provided. 
This necessitated allowing users to create a new ToolBook lesson. This task of providing an opportunity for 
practice proved to be a distraction. Users were not familiar with ToolBook and were not motivated to learn it 
since they would be using another authoring language. 
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The GAIDA design team eventually dropped that requirement, which meant that the subsequent 
GAIDA version was more like on-line tutorials provided with many software packages. The revised version 
with a checklist procedure was then tested at an Air Force Technical Training Center with six instructional 
designers. The results were positive in that all users indicated that the guidance was useful and instructive and 
could be applied to their design tasks and implemented in the various languages they had available (Gacne, 
1992). 

Figure 3 depicts the general architecture that has been adopted for GAIDA. Once the user has selected 
a case thought to be similar to the development task at hand, that lesson is loaded into memory and the 
elaboration of the nine events continues. The outer window provides a generic explanation of what it means to 
gain a learner's attention. There is additional guidance for those using multimedia which is provided via the 
"ICW" (interactive courseware) button. The "notes" button allows users to keep notes that could be used as a 
draft storyboard. The inner window provides first a generic view of what could be done in this type of lesson 
and then a textual description of what was done. The user can put the system into case-mode by pressing the 
"SAMPLE LESSON" button. 













>* 


Y Event 1 : Gain Attention 









Before beginning any lesson, you should make sure that your students are paying attention. Plan to 
direct their attention to the content of the lesson, especially when you determine that they are not 
focused on the material. This is the purpose of Event 1. Of course, it is important to match this event 
to the content that will follow. 

In a classroom, you may determine that your trainees are already paying attention, and thus leave 
out an explict attention grabber. With a 
computer based lesson, it is hard to assess 
if the user is paying attention, so you should 
always include an attention grabber. 



Rapid stimulus changes, perhaps using 
animation or video clips, is one way to gain 
attention in computer based lessons. 
Another approach is to present intriguing 
problems or novel situations designed to 
appeal to the trainee's interest and curosity. 



Pulmonary Function Check 



[5 fs !7 



For procedural learning, you may want to 
gain atention by showing the procedure to 
be carried out. Another effective device is 
to show drastic consequences of NOT 
carrying cut the procedure correctly. 

To gain attention in this sample lesson, an 
animated image of a man blowing into a 
tube is presented. 



f&lyfog { [Qo^to S^m^e Lesson | 



Figure 3. GAIDA event elaboration. 



Figure 4 depicts the case-mode, which is a part of the actual lesson. GAIDA can be executed entirely 
in case or lesson mode so that users can view exactly what students see. The expected mode of use is 
guidance mode, with the opportunity to view the lesson directly as desired. Figure 4 actually shows a text 
screen from the lesson that comes up after an animated sequence of a patient breathing into the spirometer. 
One of the hypotheses that we are testing is the utility of providing the on-line capability of switching back 
and forth between a lesson/student view and a guidance/designer view. Data collected to date from more than 
40 users indicate that this mode switching feature is one of the most frequently used features in GAIDA. In 
fact, it is used more than the note-taking capability which is a bit of a surprise. 
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Pulmonary Function Test 



jO .y- v/" ,£&g ^ Safl &g ^ ss: & -J- 



The man is taking a pulmonary function test. This 
test is used to measure lung volume and function. 



"V ■ a** 



at' • 



A measure called Forced Vital Capacity and another called 
Forced Expiratory Volume are obtained through the use of a 
spirometer. After inhaling, the patient makes a forced expiration 
through a tube connected to the machine, this produces a graphic 
record. 



A. 

4 » 




Figure 4. GAIDA case mode (lesson view). 



Two additional cases were added after the procedural cases were refined in order to provide a 
minimum repertoire of cases to support a novice courseware designer (see Figure 2). At this point, the 
GAIDA architecture has stabilized and GAIDA is now easily extensible. New example cases can be added as 
they are validated and become available. The guidance which elaborates the nine events and the specific cases 
can easily be changed to accommodate specific practices at various sites. In short, GAIDA is well positioned 
to grow into a much more robust on-line design advisor with an elaborate case base. Four additional cases are 
now under development which will extend GAIDA's domain of applicability into academic subject matter as 
well as into advanced multimedia (digital audio and video). 

Evaluation of an Instructional Design Advisor 

The evaluation of GAIDA has revealed a number of interesting findings: (1) users were confused by a 
completely open hypermedia tutorial on ISD tested in conjunction with the first GAIDA version; (2) users 
were able to follow and benefit from the structured (restricted movement) hypermedia approached 
incorporated in GAIDA; (3) controlled studies of novice instructional designers (SMEs in this case) showed 
that courseware developed using GAIDA's guidance was gener. ,ly effective and superior to that developed 
without the benefit of this guidance (Gagn£, 1992). 

It is worth noting that evaluation of an automated instructional design advisor is somewhat 
complicated. First, there must be an appropriate set of evaluations for the courseware produced using a 
performance support tool such as GAIDA. That is to say that the lessons and courses must be instructional^ 
effective and appropriate measures must be established to document this fact. However, the primary locus is 
not on the instructional intervention between learning environment and learner,, as in traditional evaluations. 
The primary focus in on the intervention between the novice designer and the automated instructional design 
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effective instruction in less time than some baseline standard?). The system must also be evaluated with 
regard to instructional utility (Was the system helpful in creating meaningful interactions for learners and did 
it help advance the novice into a more expert-like state with regard to instructional design?). These are not 
simple questions and we do not pretend to have the answers. We can say that our evaluations are continuing 
so that we can better address such questions. 



Our conclusions are that: (1) the content of the revised ISD model is useful; (2) the practice of 
following that model is time-confusing and stifling; (3) on-line instructional design cases are useful to novice 
designers; and (4) restricted hypermedia systems are appropriate for introducing novice designers to the 
complexities of instructional design. We believe that as additional cases are added to the GA1DA case base 
along with the appropriate instructional commentaries that it will be possible to accelerate the performance of 
novice courseware developers so that they will perform as advanced apprentices. The potential cost savings of 
this kind of performance enhancement are enormous. 
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Abstract: The present knowledge society places challenging learning requirements 
on all organizations. A generic example of knowledge scripting and profiling in a 
customer service organization is introduced. The example proceeds to analyze the 
hidden knowledge-intensive activities and associated concept hierarchy that underlies 
the design of knowledge transfer from experts to points-of-use. Aspects of dividing 
knowledge between workers and support systems are discussed and some roles of 
computer-based educational systems in the transfer process are identified. 



In the Knowledge Society Learning Is More Important than Ever Before 

We have entered the knowledge society - although we do not yet know the full implications (Btihme et al, 
1986; Senge, 1990; Wiig, 1991; Drucker, 1993). However, we know tbat a larger percentage of our working 
population than at any earlier time are knowledge workers - much over 50%. These workers are increasingly 
engaged in sophisticated knowledge-intensive activities. We see that the quality of their work products and 
services increases, although - because of ubiquitous competition - these improvements often do not command 
noticeable increases in prices. These are indications of desirable societal progress - as we normally think of it. 
However, as a consequence, everyone in society is constantly asked to "do more with less." 

On the individual level, the increased requirements for knowledge workers to handle more sophisticated 
knowledge-intensive activities result in needs for greater competency and expertise - Greater knowledge. To 
respond to these needs, we must prepare workers to a greater extent than ever before. They need to learn more. 

On the organizational level, the increased requirements lead to aconstant need for all organizations to 
change. As Garratt states: "For any organization to survive and have a chance of growing, its rate of learning 
has to be equal to, or greater than, the rate of change in its external environment!" (Garratt, 1990) 

In the knowledge society learning organizations must acquire knowledge on two levels. First, they need to 
learn to perpetually redesign their products, services, and business processes to provide acceptable deliverables to 
the market place at reasonable prices. Second, organizations need to transfer sufficient knowledge to their 
workforce so all can act intelligently and competently and perform the required knowledge-intensive tasks 
proficiently - with ease and satisfaction. 

What Can the Learning Organization Do? 
A Challenging Scenario - Transfer Knowledge from Expert Areas to Points-of-Use 

A large service firm has a central pool of experts. They work with difficult customer-related issues. These 
experts a^o develop new perspectives, judgments, and approaches to deal with problems and they work with 
outside experts to continually improve available knowledge. They are responsible for supporting customer 
representatives in over a hundred field locations whenever difficult situations occur. Over time, as the 
organization grew, this led to thousands of daily telephone calls and e-mail messages, most of which would not 
be required if the field reps were provided with appropriate knowledge. Unfortunately, the experts* ability to 
help has often been hampered by not having as complete information as the fields reps. Overall, the need to 
refer to the central experts has been disruptive to customers and very costly - it is not desirable at all. 

To improve customer service and reduce costs, the firm decides to transfer as much knowledge as practical 
from the central experts to the field reps. The reps are well trained in the basics of their work and the major 
questions are to determine how the additional knowledge they need can best be transferred to them. 
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The firm undertakes in-depth analyses of the knowledge-intensive scripts and activities for the most 
important (most frequent and highest value-added) tasks. These analyses identify the knowledge required to 
perform competently. The results determine the knowledge reps need to possess in their minds and the 
knowledge that can be provided by computer-based or other support systems. The resulting knowledge transfer 
program includes several modes: education (rather than training), knowledge-based and conventional decision 
support systems, and several reference documents - some paper-based and others available in electronic form. 
Some of the less frequent and more difficult situations still need referral to the central expert pool which has 
been reduced as many of the experts are transferred to demanding positions elsewhere. The pool now also 
assumes greater responsibility for benchmarking and building new knowledge where that is required. 

Some Aspects of the Learning Organization 

The example in the above scenario is typical of organizations that wish to learn actively. As outlined 
below, these organizations focus their top management's interest on long-term business implications of having 
better knowledge. To them, knowledge becomes one of the most important competitive tools they have. 

The learning organization in the above scenario is proactive. Its management agrees that the organization 
needs to manage knowledge effectively and be clear about how knowledge is built, managed, and used. It 
considers the business implications expected from good knowledge management. Management particularly 
focuses on both first and higher order benefits and costs. It conducts in-depth qualitative impact analyses to 
ascertain that it decides to do the right things. Frequently, such cost-benefit analyses are based upon senior 
management's perceptions and beliefs and good business sense rather than on strict quantitative analyses of first- 
order tangible effects. 

To understand and communicate knowledge-related mechanisms, management has adopted several models. 
The "knowledge creation wheel" (Figure 1) is a working explanation of the basic knowledge building and value- 
adding process. Management plans for the organization to continuously build its knowledge base and has 
adopted the "knowledge spiral" (Figure 2) as its representation for this concept (Ikujiro, 1991). As a measure of 
how fast the organization learns, it is in the process of identifying representative cycle times for new concepts 
and knowledge to emerge, be perfected (built), organized, and put to practical use to create value-added. 




Apply Knowledge 



Exploit Knowledge 



• Perform Kaowkdjc Work - 





• Auiomile Routine Work - 
Support Knowledge Worker* - 
■ Deploy toteUigeot Agcute - 



SeUAdviM A Practice* 
• Truufcf Technology • 



Figure 1. The Basic Knowledge Geation Wheel. 
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Figure 2. Progressing Up the Knowledge Spiral. 
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These broad concepts provide a begin- 
ning understanding of some underlying 
mechanisms. Management realizes that it 
needs to identify actual sources and 
handlers of knowledge as it is transferred 
to the points of use. In particular, it is 
important to identify where knowledge 
resides in the different transfer stages and 
who is responsible for the different 
functions. For that purpose, a high level 
knowledge flow model (Figure 3) with 
more than 30 flows has been identified. 

For transferring knowledge to the 
customer service representatives in the 
field, five different knowledge sources 
were identified as is shown Figure 3. 
Three separate functions were identified to 
be responsible for compiling and 
transforming the knowledge to be 
transferred. The existing "training" 
department has obtained additional 
responsibilities to develop and deliver 
modern "educational" programs. A new, 
separate, multi-disciplinary entity has 
been created in partnership with Human 
Resources, Training, and Management Information Systems. It is responsible for knowledge acquisition and 
analysis, and for organization, compilation, and validation to ascertain that the knowledge base is properly 
represented, well structured, and contains only acceptable facts, concepts and perspectives, judgments, and 
approaches. 

Dissemination of knowledge is performed by five separate functions as indicated in Figure 3. Each of these 
has many different knowledge transfer modes at their disposition. The modes are chosen based on the particular 
requirements of the types of knowledge and the way work is performed at the points-of-use. For example, some 
rarely needed and relatively simple conceptual knowledge is delivered to the field as part of paper-based manuals 
in addition to having been treated in the computer-based educational system. Frequently needed and more 
complicated knowledge is taught in the form of concepts and principles while some of the related details are 
provided through computer-based knowledge-based support systems. 

Detailed Analysis of Knowledge-Intensive Work 

The knowledge required to deliver quality work is analyzed by knowledge scripting and work function 
profiling (Wiig, 1993). Content for knowledge analysis is acquired through interviews, expert and average 
worker observations, and simulations where novices perform real-life tasks assisted by experts. Scripts with 
knowledge-intensive steps are identified for specific major tasks and are summarized as in Figure 4. All steps 
may not be required for each task, and different tasks may require execution of steps in different sequences 
depending upon the actual situation and the worker's associations and routines for what to do next. Knowledge- 
intensive activities for each step are identified and the knowledge needed to perform them proficiently is made 
explicit. As part of this work, major concept hierarchies (see Figure 5) are elicited from the experts to identify 
the conceptual basis for mental processing and judgments. 

A major observation by many who have previous experience in "requirement analysis" is that most of the 
knowledge-intensive activities performed by competent workers are hidden from view and not visible or 
observable by the casual onlooker. It requires considerable expertise to discern and identify these activities and 
determine their content and function. It is invariably discovered that experts perform many activities very 
quickly - almost automatically - compared with less competent workers. The quality of the experts 1 work is 
also often much better as a result of comprehensive understanding and judgment 
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Figure 3. Examples of Internal Knowledge Flows in the Knowledge Organization. 



When we deal with in-depth knowledge and 
understanding and requirements for performing 
knowledge-intensive work, we need to understand 
that we must distinguish between knowledge at 
different conceptual levels. They are: (1) Goal- 
setting or idealistic knowledge; (2) Systematic 
knowledge; (3) Pragmatic knowledge; and (4) 
Automatic knowledge. These conceptual levels 
reflect the degree to which we have internalized the 
knowledge and how we apply it to the situations 
we face. We often have intuitive comprehension 
of goal-setting knowledge but use it io direct our 
thinking and activities. We have deeper compre- 
hension of systematic knowledge and use it to 
understand "how the world works." We use prag- 
matic knowledge to make conscious decisions. 
Automatic knowledge we have internalized to the 
point that we use it without thinking. We must 
also distinguish between knowledge of different 
types: (1) Factual knowledge; (2) Conceptual 
knowledge; (3) Expectational (judgmental) knowl- 



Verify 
Caller 



r^orm Picture 
of 

Caller 
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Identify 
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Figure 4. Simplified Knowledge-Intensive Script Steps. 



edge; and (4) Methodological knowledge (Wiig, 1993). 

The Knowledge, Skills, and Personal Characteristics Profiles 

Scripting and profiling lead to knowledge, skills, and personal characteristics profiles. The profiles are 
developed in two ways: (1) From the specific knowledge components that are identified as part of the knowledge 
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analysis; and (2) From in-depth discussions with experts, managers, and others who have professional insights 
into what is required to deliver quality work in the target function. Profiles for the example scenario are 
presented in Figure 6. They are used to portray the proficiencies that are required - or that individuals mi?ht 
actually have. Gaps are readily identified to guide development of educational programs or determine wface 
individuals or groups may benefit from knowledge transfer. 

The knowledge profile denotes the different areas of in-depth knowledge and understanding that are 
needed to deliver quality work and that can be developed through education or other modes of knowledge transfer 
that facilitate deep learning. The skills profile denotes particular skills that individuals need to work 
competently and which may be improved by training. The personal characteristics profile denotes those 
personal qualities that are required. The latter characteristics often cannot be easily changed. 
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Figure 5. Concept Hierarchy for Developing the Mental Picture of the Customer's Situation. 



Teaching Concepts Instead of Training Facts Provides Many Benefits 

Concept hierarchies are central to identifying the details of knowledge required for competent work. In 
customer service situations we find that there are several top-level concepts such as "Mental Picture of Customer 
Situation," "Customer's Intent," and given these, "The Best Way of Helping Customers " 

In forming top-level concepts, workers build abstract, often qualitative, mental models. They assess the 
stale of the concept by use of discrete qualitative brackets such as: M The customers situation is not serious but 
competent technical help is required to correct the problem." To form broad concepts, they "chunk" lower-level 
concepts to combine simpler concepts into more comprehensive mental models. Chunking often varies between 
individuals and may reflect misunderstandings or different associations (Wiig, 1993). 

It has been found \o be relatively easy to teach concepts to people at all ages. Strong foundations can be 
provided to make it possible for learners to build understandings and judgments quickly and accurately. Teaching 
workers the foundations of underlying concepts reduces requirements for providing much of the detailed factual 
knowledge as has often been the tradition. Instead of teaching details for all products, it is possible to educate 
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workers in the conceptual principles accompanied with representative details to show how the abstract schemas 
and scripts translate into concrete conditions. The bulk of details can then be provided via such modes as 
computer-based support systems. Models exist for effective teaching of broad concepts and schemas while 
connecting them firmly to concrete and detailed examples. Such models suggests frequent switching between 
concrete cases and the underlying concepts and theory (Wiig & Freedman, 1993, Wiig, 1993). Transfer of 
knowledge is quicker, can be performed with less errors and misconceptions, and provides greater insights and 
flexibility when it focuses on transferring the underlying concepts populated with samples of representative 
factual knowledge. 
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Figure 6. Knowledge, Skills, and Personal Characteristics Profiles. 



What Can the Computer Know and What Must the Human Know? 



Knowledge workers who provide customer service often find themselves in time-critical situations that 
require that they "think on their feet." They engage in customer dialogs and verbal problem solving where the 
subject matter may span large domains. It is generally impossible to have these individuals possess all the 
factual, conceptual, judgmental, and methodological knowledge they need to have available to perform their 
routine and non-routine tasks. In most situations, the amount of knowledge - particularly factual knowledge - 
required exceeds the mental capacity of the average knowledge worker. However, as indicated above, it is quite 
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manageable to provide the workers with selected knowledge that will allow them to function competendy 

without possessing it all. . „ 

The challenge is to provide these individuals with knowledge and a work environment that allow them to at 
all times- (1) Understand the situations they encounter, (2) Plan where they wish to direct the situauons; and (3 
Have the security that they will be able to obtain whatever additional knowledge, facts, or mfonnation they will 

neCd We P ototave the option to provide knowledge workers in these situations with powerful computer-based 
support systems to assist them. When this is an option we are immediately faced with the issue, "What can the 
computer know and what must the human know?" . 

One solution is for the computer to have all factual knowledge and also have as much methodological, 
conceptual and judgmental knowledge as is practical. That knowledge may be represented in an active reasoning 
system or less desirably, in a sophisticated but passive query system such as hyper-media. The human should 
be provided with as much conceptual knowledge as possible to understand all general principles that underlie the 
situations s/he is expected to deal with. In addition, s/he must know how the factual knowledge relates to the 
concepts and how to apply all methods. The particulars of the methods can be obtained from the support system 
when required. Particulars of judgment and other specifics may also be obtained from the support system as 

needed . . 

This division of knowledge has many advantages. Educational requirements can be kept to a minimum. 
Workers can be more versatile and flexible and can change jobs as required. Volatile knowledge may be 
incorporated into the support system whenever it changes. 

Designing Knowledge Transfer Programs 

Knowledge transfer programs must be designed with great care to complement the organization they will 
serve There are numerous knowledge transfer mode options and many of these are very effective and 
sophisticated but may require considerable infrastructure. As computer-based multi-media delivery vehicles are 
reduced in price, it makes great sense to consider these as part of the delivery process. On the other hand, human 
interactions, networking, and teaming are (still) very important and need to be considered for many important 
knowledge flows. 

Computer-Based Educational System Roles In the Knowledge Transfer Process 

Transferring knowledge from its sources to points-of-use proves very valuable ~ but also very difficult and 
costly It is always expensive to elicit, organize, and structure knowledge in computer-based systems. 
However, when that is done, it becomes inexpensive to deploy it. The up-front costs of teacher-delivered 
knowledge is often negligible but in large organizations the cost of providing personal one-to-many education 
can be excessive. Other factors must also be considered. Personal real-time education may interfere with 
working schedules and may involve costly travels whereas computer-based systems may provide highly flexible 
time management opportunities. 

Designing computer-based educational systems may be quite different from designing conventional 
computer-based training systems (Helander. 1988). The knowledge acquisition required to elicit and organize 
knowledge requires higher specialization and is costlier. The design of these systems appears to be more a 
function of the particular knowledge. It is not a standard approach. In spite of these difficulties, modem 
knowledge transfer is already highly dependent upon computer-based educational systems and, as our expertise 
increases, they will become more important 

Another opportunity for computer-based education relates to the fact that we occasionally find that some 
high-performance knowledge has a very short half-life and needs to be refreshed through games, simulation, or 
other means. Computer-based educational systems, resident in workstations, are found to be very useful for this 
purpose. 

Concluding Perspectives 

Learning organizations need to become expert at building and transferring knowledge from its source to 
points-of-use. They need to lcam to deal with knowledge management and analysis in greater detail than most 
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now are prepared to do, and they need to have technical expertise to staff all knowledge management-related 
functions in the process. 

When considering how knowledge should be managed, it is imperative that senior management's 
perspectives of how and where the business should go drives the process. It is similarly important that 
considerations for potential value-added contributions from knowledge management actions be considered, given 
senior management's visions for higher order market reactions and such notions as what it means to deliver 
quality products and services into the market place. 

For a learning organization to be successful in these efforts, it is a requirement that the organization's 
culture and incentives arc changed to facilitate and promote all the activities that need to become second nature 
for all within the organization. That refers to the need to learn at every opportunity, as well as the need to share 
knowledge with others. It is clear that if the culture and incentives support the separate contributions of 
individuals rather than team efforts or the overall organization - there will be little motivation for experts to 
share their knowledge with others. 
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Abstract: As a result of [the society] moving into the information age, changes in die 
educational process as well in the workplace are inevitable. Parents students, school 
officials and business leaders are currently experiencing the effects of the mlormaiion 
explosion era on a daily basis. Upgrades and new releases of computer hardware and 
software are occurring almost before we can fully understand our old systems. Arc 
decision makers ready to make the changes necessary to prepare students with the 
competency and foundation skills (as identified by die Secretary's Commission on 
Achieving Necessary Skills) that they will need for the changing workplace? 1 his 
paper will address two trends in the instructional technology held: reiorming the 
educational process and implementing electronic performance support systems into the 
workplace. These trends are important because they have nilluence on moving Hie 
teaching and learning process to a uew evolutionary stage. 

As society approaches the 21st century, technology will play an important role in the lives of adults as 
well as in the lives of children. Business leaders, school officials, parents, and students need to be aware ot die 
impact technological change will have on the educational system as well as on business and industry 
S Achieving Necessary Skills [SCANS] 1991) • HirUiemi ore . with markets 

h^ominff more elobal and companies desperately striving to stay ahead, jobs ot the i mure will demand more 
SBlSK^ & AbLeneVs; SCANS, 1991 ; Work Force 2000, 1988). Dunng the 
comine years jobs that require skills with information-processing technology will be ol the norm radier than a 
rarity (Naisbitt & Aburdene, 1991). To prepare students for a work force that is different lrom their parents 
Keieeluth (1992) argues that the educational system requires moving horn an industrial age model towards an 
infonnation-age model that views the educational process as the integration of learning tasks, widi the teacher 
as a coach or facilitator using electronic technologies as tools within the classroom. 

The mastery of reading, writing and arithmetic will no longer be enough to propel students to success in 
die work place. According to the SCANS (1991) report, researchers identified live competency skill areas mid 
three foundation skill areas that are necessary for success in the workplace. 1 he live competency skills, as 
defined by SCANS (1991) are: 

1. Resources: Identifies, organizes, plans, and allocates resources, 

2. Interpersonal: Works with others, 

3. Information: Acquires and uses information, 

4. Systems: Understands complex interrelationships and, 

5. Technology: Works with a variety of technologies. 

In addition, included in the SCANS (1991) report is a description of the three-part foundation skill 
requirement. They are: 

1 Basic Skills' Reads writes, performs arithmetic and mathematical operations, listens and speaks, 

2 Thinkinc Skills: Thinks creatively, makes decisions, solves problems, knows how to learn and 

3*. Personal Qiudity Skills: Displays responsibility, self-esteem, sociability, sell -management, integrity and 
honesty. 

Will high school or college graduates be able to master the competency and foundation skills upon graduation? 
This question focuses on how society is changing and as a result the needs of students who are preparing to 
enter the workforce have changed. 
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Education is our passport to the future, for tomorrow belongs to the people who prepare for it 
today. 

Malcolm X 

Educators are seeking new ways to ensure that students have the appropriate skills on a local and national 
level. On the local level, school officials are re-thinking the design of schools for the information age 
(Molenda, 1992; Rcigcluth, 1987) and on the national level the Bush administration announced America 2000, 
a new education strategy, designed to provide national educational goals for the nation's schools (Elcy, Foley] 
Freedman & Scheel, 1992). The fundamental purpose for these initiatives is to empower students to tackle the 
challenges facing them in "the real-world". 

Employees can no longer believe in job security (Naisbitl & Aburdcne, 1985) or that a college degree 
alone (without constandy upgrading ones' skills) will propel him or her in the workplace. Employment lor 
many people, currently and in the future, requires a commitment (whether embraced by the employee or imixxscd 
by the employer) to change and life -long learning. Naisbilt and Aburdene (1985) argue that "in the new 
information society where the only constant is change, we can no longer expect to get an education and be 
done with it. There is not one education, no one skill, that lasts a lifetime now" (p. 141). Furthermore, 
Packer (1988) argues that if the U.S. economy is to stay competitive, "twenty-five million American workers- 
will need to upgrade their basic skills during the 1990V (p. 8). This is a low estimate when taking into 
account the bases lor projections are only on the changing mix of jobs and do not include workers who need to 
upgrade their skills as a result of new technology (Packer, 1988). 

Nevertheless, will graduates be ready to face a world that is constandy changing at a rapid pace? There 
was a time when a clerical employee learned how to type and file mid would not have lo learn a new procedure 
for years- if ever. However, today die replacement of a word processor occurs almost as often as die assignment 
of a new manager. Many times, in response to receiving this new software application, the employee must 
become proficient without a decrease in productivity. This situation places the employee in the position of 
seeking the quickest and "least painful" method of training. 

By keeping abreast of the changes in society as well as in education and training, instructional 
technology prolessionals can play a key role in restructuring the educational system from its current industrial 
model to a system that reflects the information age. In addition, restructuring thai includes iiucgnilin*' 
electronic technology into the educational system can be one solution for bridging the skill »ap from hi«h 
school or college to die workplace. This restructuring movement can also be the catalyst for using electronic 
technology as a tool lo assist the leather in developing the competency and basic skills as defined by the 
SCANS (1991) report. Furthermore, to bridge the skill gap for employees accessing new information 
employers are investigating electronic performance support systems (EPSS) which is a computer-based tool 
designed to assist the employed with on-demand training and support. 

The implementation of EPSS in the work place and the restructuring of the educational system are two 
important trends in the instructional leclmology field (Eley et al., 1992; Seels, 1993). These trends are 
important because Uicy represent moving the teaching and learning process in our schools and on-the-job to a 
new evolutionary stage. 

A Proactive Mindset to Change 

Systematic Change Process 

Change, whether planned or chaotic, brings on stress and anxiety. Moreover, unplanned change combined 
with minimum involvement from the people affected by the change, produces resistance (Burke, 1982). 
Furthermore, Burke (1982) argues that "what people resist is not ch.uigc but loss, or the possibility of loss" 
(p. 52). This loss is generally one of two kinds: loss of the known and tried or loss of personal choice 
(Burke, 1982). Rickleman and Henk (1989) argue that change does not have to be unsetdinj» if everyone 
involved take a proactive stand. 

By taking a proactive stand, we shift from asking "What is going to happen to us in the 
1 uture?" lo "What can we do to create the kind of future we want?" (Rickleman & Henk, 1 989, 
p. 175). 

Kurt Lewin, a psychologist and educator, was one of the earlier researchers to take a proactive stance for 
change. Lewin argued that a key principle to organizational change is a three-phase change process "unlVee/iim 
moving, and relrcc/jng." In trying to imagine die phases required for change Lewin used the metaphor of 
melting ice. Uiifree/jng means reducing the negative forces through new or discontinuing information, which 
is the function of diagnosis. Changes in attitude, values, structure, feelings, and behavior incorporate the 
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moving stage and refreezing means reacting a new status quo with support mechanisms to maintain the desired 
behavior (Weisbord, 1987), 

Many systematic change models have evolved from Lewin's model of change (Burke, 82). A key issue 
with Lewin's model is the involvement of everyone associated with the process. In addition, Lewin states 
"people are likely to modify their own behavior when they participate in problem analysis and solution and 
more important they are likely to carry out decisions they have helped to make" (Weisbord, 1987, p. 89). 

Educational reform is a hot topic in the school system as well as in all levels of government. Recently, 
the reform movement has been shifting from a disjointed incremental effort towards more of a Gestalt 
perspective, where the focus of change is on the whole educational system (Bagley & Hunter 1992; Campoy, 
1992; David, 1991; Pearlman, 1989;Raywid 1990; Sheingold, 1991). 

David (1991) argues that "restructuring presumes the goal for educational system is not simply to eatcli 
up to the world, it needs the capacity to continue to evolve as the world continues its rapid pace of change" (p. 
77). To reach this goal Reigeluth (1992) identifies "Ernest Boyer (1983), John Goodlad (1984), Theodore 
Sizer (1984), Lewis Pearlman (1987), Ann Lieberrman and Lynne Miller (1990). Albert Shanker (1990) mid 
Banathy (1991, 1992)" (p. 1) as advocates for systemic change in education. Reigeluth (1992) further argues 
that true educational reform has occurred only once in our nation's history- when the industrial assembly-line 
model, which is currently in place today, replaced the one-room schoolhouse. This paper will not address all 
the issues associated with restructuring the school. However, the author will focus on a small portion of the 
educational reform movement that considers integrating electronic technology into the classroom jmd the uses 
of EPSS as a tool for promoting training and support in education or business and industry work settings. 

Integrating Technology into the Educational Process 

Restructuring the educational system is not an easy task. It can be a very complicated and long tenn 
project yet, the results of reform can be fruitful mid enriching especially if decision milkers do not view 
electronic hardware and software as the sole answer for entrenched ills, A decision maker who believes 
integrating computer-based technology (into the school system) requires simply supplying teachers mid 
students with additional new hardware and software is focusing on a "device-driven" learning enviroiunent 
versus a "student-centered" environment. Bagley and Hunter (1992) cited Rockman who cautions "policy 
makers against 'tcchnohype 1 , which he describes as the efforts by advocates and vendors to sell technology as 
the one and only answer to restructuring the school system" (p. 22). Sheingold (1991) further argues that 
"computer-based technology has been brought into schools during the past decade largely because the 
technology was seen as important in and of itself- because it was an increasingly central component of die 
world of adult work" (p. 77). Technology is more than just electronic devices. Elcy et al. (1992) define 
technology as applying scientific principles to solve practical problems- a process that deals with problem 
solving. This definition decreases the emphasis from specific devices and places it on solving the problem 
(Wager, 1993). For example instructional design and teaching and learning methodologies fit the definition of 
technologies, however they are forgotten examples. The author suggests that the lessening of "tcchnohype" emi 
occur by informing decision makers that electronic technology is "a systematic blend of people, materials, 
methods and machines" (Eley ct al., 1992, p. 27). 

More important, the integration of computer-based technology into the educational process requires 
decision-makers to re- think tiieir opinions on several topics- for example their views on: 

1. The role of the teacher in the classroom as well as their views on teaching and learning, 

2. The organization of student desks and computers in the classroom, 

3. The life-cycle and maintenance of computer hardware and software. 

In addition Collins (1991) identified, from the literature and observations in schools where teachers are using 
computers, several major trends on how computer technologies have an affect on the classroom. First, Collins 
(1991) argues that the role of the teacher will move from a lecturer to a facilitator mid coach who actively 
engages students in long-term computer projects. These projects may simulate relevant "real -world" problems, 
where students are actively participating in arriving at solutions. 

The classroom will shift from desks placed in rows where the whole class participates to desk arranged in 
small groups. This new classroom arrangement, along with the teacher's encouragement, can provide an 
environment where students can focus on cooperation and collaboration. Lastly, the author suggests that by 
exposing students to electronic technology used as a tool or resource for the teacher students will have an 
opportunity to practice the foundation mid competency skills as describe by SCANS (1991). 
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Sheingold (1991) provides several recommendations that may assist in the change process that includes 
integrating computer technology. They are as follows: 

1. Bring technology and learning to the same 'table 1 when restructuring is being planned, 
2 Reconsider how technology is organized in the district and finally, 
3. Work towards teacher expertise in using a critical mass of technology. 

Instructional technologists are also facing issues on how to support employees, such as teachers, 
administrators and business managers, when many school systems, universities and businesses are facing huge* 
budget cuts and downsizing efforts. As a result of these changes, employees are being called upon to "do more 
with less" and decision makers are looking towards instructional technologists to investigate ways electronic 
technologies can increase productivity with a leaner workplace and fewer people available for employee 
support. 

Currently, employees may receive their formal training off-the-job in instructor- led courses or by 
computer-mediated training (interactive video, computer-based training, or a combination of both), however, 
when the employee returns to the actual job, training and support may consist of human interaction, non- 
centralized reference manuals or company documentation. Problems arise when employees need specialized 
training but none is available until several weeks or months later. Uncertainties in die employee's ability also 
arise when he or she is trying to complete an assignment and need specific support but receive either too much 
information, conflicting information or no information at all (Brcchlin & Rossctt, 1991). 

Employers can no longer ignore employee training. Decision makers are slowly realizing that training is 
a factor in ensuring that the workforce is productive. It is only after the elimination of training programs diat 
decision makers sometimes realize its importance. 

'Training is like rowing against the current. Once you stop you are dragged downstream." 

Jozef M. M . Ri tzen 

As a result of these problems, decision makers are entertaining the idea of providing employees widi software 
applications that provide immediate support and on-the-job training (OJT) from the employee's desktop 
computer at the specific moment of need. 

The development of electronic performance support systems (an integrated electronic system that provides 
training and support at die moment of need for the employee) is a trend in instruction^ technology that can 
bring training and employee support to the desktop computer. In addition, with the introduction and 
advancements of tools such as: relational and multimedia databases, computer-based training, expert systems 
and on-line references, the foundation for developing electronic performance support systems (EPSS) is in 
place. 

What is an EPSS? 

Since the recent introduction of EPSS into business and industry, leaders in the field are still trying to 
develop a working definition associated with this tool. Raybould (1990) describes an EPSS as a "computer- 
based system that improves worker productivity by providing on-the-job access to integrated information 
advice and learning experiences" (p. 4). Whereas Gery (1991), a leader in the field of EPSS, defines an EPSS 
as "an umbrella concept that includes any of a variety of performance support interventions delivered on 
computer to the worker on the job at die time of need" (Clark, 1992, p. 36). 

In describing an EPSS there are two major areas to consider, the content and die components of the 
system. To identify the contents of an EPSS, Gery (1991) uses the term "Infobase" which is the collection of 
information the employee will inquire into, access, or have presented to him or her when accessing die system. 
For example, the information located in a text relational database, multimedia database, expert system or on-line 
reference system are the type of data the employee could select from within an infobase. When manipulating the 
infobase, die user interacts with the components of die EPSS that can include die following: 

A range of support mechanisms and software tools including an advisory system to help in 
instructing or executing tasks and decision making, commercially available software programs, 
organizational specific application software, special purpose software utilities built especially for 
use within die EPSS and other interactive capabilities (Gery, 1991 , p. 42). 

Components of a complex EPSS can, ideally, include a combination of hypermedia databases, expert systems, 
modular interactive training, a dynamic maintenance system, as well as other interactive software support 
applications, whereas, a basic EPSS may include only a database mid an on-line help system (Gcber, 1991). 
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The common feature found in both a basic and complex EPSS is the ability of the application to provide 
information to the user at the moment of need (Scales & Yang, 1993). 

EPSS in Educational Process 

Researchers are investigating and developing EPSS and other support tools in academia as a way to 
increase productivity in the workplace. Merrill (1993), Gustafson (1993), and Dick, (1993) are advocates lor 
increasing productivity as a result of decreasing the instructional development time by using an hi SS. 
Educators from Florida state looks towards an EPSS to increase productivity by providiug an integrated 
information and learning system for staff involved in special education. This project is part of Honda State 
School year 2000 initiative (Huestia, 1993). In the mist of companies as well as academia downsizing 
employees are being called upon to "do more with less". To assist employees with this change EPSS arc being 
designed to bridge the training and support gap. 

Decision makers who are not ready to invest in an EPSS, may consider embedded interactive training, as 
a solution for some OJT needs. Embedded training, as defined by Andrews (1991), is the integration ol 
multimedia training applications into a workplace tool. EPSS and embedded training applications are moving 
the training from out of a classroom onto the employee's desktop computer. In to prepare lor the coming 
chances in workplace training, managers, software developers and instructional designers, will be laced with 
seeking a clearer understanding of the EPSS development process. More specifically, decision milkers must 
addressing the issue of moving the development of training from the end to an earlier software development 
stage Therefore EPSS or an embedded twining application will change the way they receive training on-the- 
job In addition, individuals involved in developing software must work together in re-tlunking how to 
incorporate the development of training applications as part of the initial software development process. 

Changes in Software Development Process 

In order to take full advantage of trends such as the computer-base training component of an EPSS, or 
embedded training, changes in the software development process are necessary. Traditionally the development 
of training and support documentation have not been an integrated part of the software development process. 
Howell (1992) provides a summary of six methods used for general software development projects. Of these 
six techniques, none address training and support as part of the development process. However, advances m 
computer hardware and software technology have made it possible to integrate embedded training and support 
within the software development process before the completion of the project. 

The author suggests that to effectively develop the training component of an EPSS, embedded training or 
even non electronic support materials, the traditional software engineering model should be modified. Figure 1. 
displays a software engineering model that accommodates the pjirallel development of the application software 
process user interface process, and has been modified to incorporate the parallel development of Induing and 
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user support materials. This abbreviated software development model is one way to displays a general diagram 
of each of the three development processes. It is important to note that throughout each process efficient and 
effective communication must occur between each development group. By moving the training and support 
process from the end of the software development process the user will be able to test a final system complete 
with training and support materials. In addition, this is one suggestion of a software development model for an 
EPSS. Furthermore, this change would incorporate the price of training in the overall cost of the system 
whereby eliminating the purchase of training as an afterthought or as part of the software bundle that is "nice to 
have". 



Summary 

Educational reform and the development of EPSS are occurring as the result of the changes in the skills 
and knowledge needed for the workplace today. Moreover, the implementation of educational reform and EPSS 
in instructional technology will require following a systematic change process that looks at the entire 
organization for change and not just single components. Restructuring the educational system is not an easy 
task therefore it requires advocates for change committed to an on-going and sometimes difficul t process. Any 
advocate for systematic change ii\ education must first question him or herself on their views of teaching and 
learning and the role of electronic technology in the classroom. Having an advocate for change who is 
uniformed could potentially promote technology as a cure all versus ;is a tool for supporting teachers in their 
leaching and learning process. More important, to successfully integrate computer hardware and software into 
the teaching and learning practice, advocates for change must prepare teachers and administrators not in just 
"how" the computer works but also in how the teacher and student can learn "with" the compute using it as a 
tool or a resource. This concept calls for a change in the role c r the teacher, classroom structure and social 
structure. 

Instructional technologists are also being called upon to develop EPSS. The idea of supporting an 
employee with information, advice and training at their specific moment of need i s attractive to employers who 
arc seeking to increase productivity in the mist of downsizing and budget cuts. However EPSS is not a cure 
all solution lor every training situation and does not necessarily mean eliminating additional training before or 
after using the system. 

Lastly, our society has moved from the industrial era to the information and service age. Decision makers 
whether in the work place or academia, will require involvement from their entire organization to prepare our 
luture graduates with the appropriate skills for a changing society. Arc business leaders, school officials 
parents , and students, ready for this challenge? 
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Abstract A cognitive rationale is proposed for the utilization of electronic performance 
support systems (EPSS) to aid in human information processing, thereby increasing 
levels of skilled performance. Current cognitive theory in the areas of attention and 
memory is reviewed to describe how skilled performance is developed and maintained 
in a performance-oriented context Three types of performance limitations and their 
respective cognitive explanations are described to elucidate the information processing 
limitations inherent in real performance contexts: (1) Performance requiring large 
amounts of factual knowledge (high demands on long-term memory); (2) Performance 
requiring proficiency of skills infrequently utilized on the job (high demands on 
declarative and procedural knowledge that was never adequately processed or that 
decayed); and (3) Performance requiring simultaneous processing of large amounts of 
information (high demands placed on a limited capacity working memory). A 
performance intervention technique in the form of an EPSS is provided for each 
limitation. 

Suppose you are a new employee in the customer support department of a software manufacturer. 
You have just received over 100 hours of training including role-playing in a simulated performance 
environment to help prepare you for the myriad of possible questions you will encounter once on the job. Not 
only are you expected to demonstrate competency, perhaps even proficiency, within a few weeks in the many 
areas of customer support, but you are also expected to quickly adapt to rapidly changing software 
specifications. Even if you have acquired the declarative knowledge (factual knowledge) and procedural 
knowledge (knowledge about "how" to perform the task), you may be required to process too much information 
at one time to perform skillfully. This is considered an overload of working memory, the temporary and 
limited-capacity memory store where it is believed information processing takes place (Baddclcy, 1986; Klapp, 
1987; Miller, 1956). Another performance limitation may be that you were adequately trained to perform a 
particular skill, however, the skill may be required so infrequently that the skill decays. If human performance 
increases in almost any cognitive skill with the amount of consistent practice of that skill (Schneider & Fisk, 
1982), then it should be reasonable to ask the following question: How could enough practice possibly be 
provided in prior training to develop competent skills for the large variety of problem-solving situations that 
employees will face? 

There arc some practitioners in the performance technology industry that suggest that the types of 
performance problems mentioned above are not the result of inadequate training, but rather the result of using 
an old training "paradigm" in a new business climate (Geber, 1990; Gcry, 1991). The old training paradigm 
prescribed the development and implementation of training programs if employees did not have the required 
knowledge and skills to perform adequately. A new business climate is emerging that is causing performance 
technologists to question the value of this approach to performance problems for several reasons. First, 
products and services are changing more rapidly today than ever before. Employees must keep up with ' 
constantly changing information if they are to remain competent and if their employers arc going to remain 
competitive. As a result they must receive training much more frequently than before. Secondly, employees 
are being more frequently placed into jobs for which they were not adequately trained. This is a result of the 
need for businesses to fill positions more quickly to survive in highly competitive markets. Finally, 
businesses are being required to operate with a more diverse work force. Businesses arc expericne ing a growing 
need for being able to train a workforce varying in educational, cultural, and socio-economic backgrounds. 

What Gloria Gcry (1991) and others in the performance technology industry have called for to meet 
these challenges is a new paradigm that expands the arsenal of performance intervention techniques to include 
innovative approaches to old and new training problems. In her book, Electronic Performance Support 
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System Gery (1991) outlines the major "paradigm shifts" required of performance technologists for the 
successful adoption of these new techniques. At the heart of this paradigm shift is the movement away from 
thinking that all training must lake place as a formal training event to a realization that training is an on-going 
process and is most meaningful when it occurs in the context of performing the task. 

One technique is to offer employees training in the time of need, in the right amount, and in the right 
form to compensate for the knowledge gaps and skill deficiencies believed to be responsible for performance 
problems. This technique is commonly referred to as performance support According to many proponents 
(Gebcr, 1990; Gery, 1991), performance support embraces the notion of learner-centered instruction, one ol the 
essential principles of adult learning theory. Moreover, learners must be involved in their learning if they arc 
ever to develop competency. 

Competency is usually viewed as having the necessary knowledge, skill, or experience lor some 
purpose. Highly competitive organizations, like the software manufacturer that must compete with other 
manufacturers in the area of support service to survive market demands, need support personnel that arc more 
than just competent They need personnel that are proficient Proficiency goes beyond mere competency in 
that proficiency demands that an employee function as an expert While a competent employee has the suitable 
knowledge skills, and experience to perform a majority of the tasks at an adequate level and in a sufficient time 
frame the proficient employee has a vast repertoire of knowledge and refined skills that can be executed with 
relative ease. This level of expertise is the central goal of this new paradigm that seeks to provide employees 
with the necessary tools to function at this level with minimal prior knowledge, skill training, and experience. 

The new tool integral to the success of this approach is what has been coined by Gery (1991) and 
others as an Electronic Performance Support System (EPSS). In its most basic form, an electronic 
performance support system (EPSS) is a computer-based performance tool consisting of an informational data 
base an advisory system, and a tutorial/learning base composed of granular-sized pieces of computer-based 
training all linked to one another in a relational data base. With the help of an integrated graphical user 
interface (GUI) and a few clicks on a pointing device, the user can access a rich variety of media on a number of 
topics. Most EPSS's use a hypermedia system that allows the user to create their own path to any number of 
topics delivered by a combination of text graphics, animation, audio and/or video. The chief operating 
function or characteristic of this system that allows the user (i.e., employee) to solve problems due to a lack of 
prior knowledge, experience, or training is the ability to quickly access well-structured pieces of information, 
advice, and tutorial assistance just at the moment of need as determined by the user in the context of the present 
performance problem. It is this type of consistent easily accessible, user-oriented support that advocates 
(Gebcr, 1990; Gery, 1991; Raybould, 1991) claim to be the key to this new technique to enhancing human 
performance. 

The purpose of this paper is to draw from current cognitive theory to describe how skilled performance 
is developed and maintained in a performance-oriented context Skilled performance is the application of 
knowledge and skills required to adequately perform a variety of related tasks in their context Three types of 
performance limitations will be discussed and analyzed from a cognitive perspective to uncover the human 
information processing limitations that are responsible for inadequate performance. These three performance 
limitations and their associated cognitive explanations are listed in Figure 1. 



Performa nce Limitation 



1. Performance requiring large amounts of factual 
knowledge 



Cognitive Explanation 



1. High demands on long-term memory without 
adequate processing 



2. Performance requiring proficiency of skills 
infrequently utilized on the job 



2. High demands on declarative and procedural 
knowledge that was never adequately processed 
or that decayed 



3. Performance requiring simultaneous 
processing of a large amount of information 



3. High demands placed on a limited capacity 
working memory 



Figure 1. Performance Limitations and Cognitive Explanations 



The extant cognitive literature in the areas of attention and memory relating to the development and 
maintenance of cognitive skills will be examined. A theoretical framework based on this research will be 
provided for discussing how electronic performance support systems can be utilized to aid human information 
processing and thus increase levels of skilled performance in the contexts of the three performance limiuilions 
outlined above. 
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The Development of Skilled Performance 

kt ti «!1i ICh h K ^ ^!i tten describin 8 «•» various stages of cognitive skill acquisition (Anderson 1982- 
NeweU & Rosenbloom, 1981; Schneider & Fisk, 1983). Fitts (1964) stated that skill lcarni^Srs in three 
stages: (1) a cognitive stoge, in which a description of the procedure i learned; (2) an wSStoSZ in 
which a method for pcrformtng the skill is worked out; and (3) an autonomous stage, 5 which The sk.ll 
becomes more and more rapid and automatic. In analyzing each one of these stages, Anderson's ACT fAdantive 
Control of Thought) model (Anderson, 1983) wiU be use/to trace the development of Srt ^ow^T 
each of these three stages. Attention and memory theory will be discussed within the framework of an 
informauon processing model to demonstrate the cridcal role each of these cognitive components plavs in 
determining the rate and level of skilled performance acquisidon. Munponona piays in 

The Cognitive Stage 

In the cogniUve stage the learner is involved in the process of acquiring what Anderson 1 19821 riik 
dec arauve knowledge. DeclaraUve knowledge can be viewed as a set oflcts S^ZTa^Sl s^l 
t*! 0 ** 0Uf CUS ^ mer SUPP ° rt re P resentativc would include such facts as how much memory docs the 
software program need to operate, the necessary steps involved in installing the application, visual imaS of 
how various computer screens look, and other types of factual knowledge relevant o^Sng The soTSe 
This knowledge is still flexible and not committed to how it will be used (Anderson 1987) 

In Anderson's ACT model, declarative knowledge is what makes up the component known as 
declarative memory. In die context of our scenario, this is the store for factual infoSon anZL about the 
software and how to support customers. In the old paradigm there is an attempt to leach as much of £ 
nformauon as possible to the learner in hope mat they will retain enough informauon to peSorm we? once on 
the job. The new paradigm attempts to provide this informauon "along the way" as the learner (emnlovS 
encounters the need for it. In this approach the learner is able to make connections bcte^fte nZSSSn 

dccJarauye knowledge and what he refers to as procedural knowledge. DeclaraUve knowledge is knowing "ih-.i" 
about a domain whereas procedural knowledge is knowing "how". Both of these typeTo XwlcdS Tare 
important in the development of skilled performance. Procedural taowta^SS^^wSf 
representative example would be knowledge about how to convert a document from i olTsoftw^ version to a 
new software version. The old training paradigm would prescribe that the IsuS^^^^SSii It 

* rCCa " ** StCpS inV0,VCd m * C P r0Ccdure - At **> lhe SSSiKSS offer 

anenormous reference manual that contained a step-by-step procedure. The new paradigmTuggests 2 : lie 

S^^ edgC mU H lX A, appliCd consistcntl y i" *e context of the performance probleS Sed 
?££TSS e . l0 . be ac< l u ' red This is consistent with the evidence found bVresearchers (Schneider A Erin 
1977; Shiffnn & Schneider, 1977; Chase & Ericsson, 1982; Schneider & 3* 1983) £5^£iK 
increases in human performance on a task depends on the amount of consistent ffiSk 

In learning how to trouble-shoot (e.g., analyze a problem, eliminate improbable causes formulate a 
solution) customers' problems in our scenario of the customer service ftpKsm^p^^u^^lot 
actual mformauon (both decorative and pnxedural) may have direct relevance to •^JZt!S(X l JS 
It k n 2?? " &PPCar ta J? COntrast 10 ^ ncw P^S" 1 Prescription previously mentioned However 
InZrl T , PO^ss'ng" Cath^ than acquiring) that is significant in understanding the mSuencss of thi ' 
SEK An H eIcctron,c fr^^ance support system can serve as a an extension of long-te™ memory for 
declarative and procedural knowledge necessary for solving a variety of problems. The repSnSu ve can 

2SK f ° f i " f ° nnati0n whi,c 3,80 rcccivin « expertadvice from a £d?rK SXvice is 

context-specific and integrated into the infobasc. The rcpresentauve is able to practice usingtheTpSS to 2 

SJ^SSS prob,cms whi,c not sufferin8 in 11,0 cognitivc stage from 3 ^^32 - ' 

To understand how other performance limitauons arc created during skilled performance acauisition and 
how they can be mediated by EPSS's, a model of human information processing is helpSirdtSwn^r 
fi&T?? P ?T Wl ! CnCOd ^' St0rcd • 3,1(1 rclricvcd - This W k P rim arily concerfed with pweTsfng" 
nA^ 

in Autinson ana ihuTnn s (1968) model of the human informal on processing system there are three tvno* „r 
memory components called the sensory memory, short-term memory, and lolSSmm^on Anothcnerm 
used to refer to a component of the memory system is working memory. Work nTmw has been 

It °" al y th ° U8h , 1 f 38 "? m ° f *» mcmor > s y stcm whc "> active information^ pSng j£ Ice - 
analogous to a mental scratch pad (Baddeley, 1986). Working memory is limited foSp ac! t * pS £« 
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information as it must switch between various control mechanisms. Some of these mechanisms which are 
crucial to carrying out a task are controlling perceptual attention to various input stimuli, selecting memory 
strategies to facilitate short-term and long-term memory requirements, and selecting, modifying, and executing 
problem solving strategies. The term short-term memory is sometimes used interchangeably for the term 
working memory. For the purposes of this paper, the term working memory will refer to both the area where 
active processing takes place as well as the limited-capacity storage place for items of information held for 
processing. 

There are two types of working memory processing limits: (1) limited capacity and; (2) duration. 
Limited capacity is tne amount of information (number of individual units of information) a person can 
maintain in their working memory at one time. Miller (1956) determined that the average capacity for holding 
to«be-remembered items in this memory store is seven items, plus or minus two. Duration is another 
important aspect of working memory capacity. Duration refers to the length of time a set of information can 
remain active in working memory without being actively rehearsed. Peterson & Peterson (1959) demonstrated 
that a sequence of three consonants, well within the subject's tr inory span, will show precipitous forgetting 
over a 20 second period if the subject is prevented from rehearsing the material. These two types of processing 
limits can have an enormous impact on the acquisition of skilled performance as will be discussed later. 

Working memory (also referred to as short-term memory) can be compared to a more permanent store 
for facts or propositions (Anderson, 1982) referred to as long-term memory or long-term store (LTS). This 
memory component is considered to be unlimited in storage capacity, however, retrieval of information is 
sometimes difficult The mechanisms for retrieving information from long-term memory and the ways in 
which EPSS's can aid in this process will be addressed in more detail later. 

All knowledge in the ACT theory starts out in declarative form as individual propositions. A 
proposition is a basic unit of meaning that expresses a simple relationship or idea between two concepts. An 
example of a proposition for the customer service representative would be "a floppy is a disk". The 
representative's knowledge of the software, the computer, and all of the other related information acquired in 
training are stored in memory as an interrelated network of propositions, connected by linking pathways, and 
accessed by means of spreading activation. Spreading activation is a theoretical process by which one 
proposition activates another in iis pathway as determined by its strength in relationship to the retrieval cue. 

The way in which each one of these components works can be demonstrated in a scenario involving 
our customer support representative. A customer asks the representative how to select hisNher new printer for 
current use. The question is received in the sensory register (in this case the auditory register) and deposited 
into the limited-capacity working memory. Once here, probe cues are generated to activate propositions in the 
long-term memory related to the probe cue. In this cognitive stage, the novice representative is searching for 
links among propositions in declarative knowledge, or in the declarative memory component (Anderson, 1983). 
The answer is a series of steps (propositions) that together make up a procedure. Contextual information 
already in working memory concerning the problem is also encoded in the probe cue. The probe search can be 
viewed as a scries of cycles (Shiffrin, 1970): (1) generate probe information and place it into working memory; 
(2) use the probe information, along with the general contextual information presently in working memory, to 
activate associated information from long-term memory; (3) scan the search set and choose some relevant 
information for appropriate decision making; (4) decide whether the appropriate information has been found and 
whether the search is over. The success of retrieving the correct information from long-term memory relies on 
a variety of factors. The probe cue is made up of probe information generated by the representative as ; as 
transient information like context. Th£ context in which the information was originally stored is of primary 
importance in determine retrieval success. If, for example, the trainer made a particular reference to a 
mnemonic device for remembering the steps, such as a word whose letters stood for each step, the link to that 
information in long-term memory may be activated by hearing or seeing that word. Efficient retrieval will 
occur when the transient context in working memory at the moment of test has not changed considerably fr^ 
that during storage. This of course depends on the knowledge base and structure of the customer service 
representative. A novice has fewer associative links to the missing information than does an expert. Also, 
practice helps to strengthen the pathway between propositions resulting in faster and more accurate retrieval 
rates (Anderson, 1987). Providing access to retrieval cues and thus enhancing long-term memory retrieval is 
one way in which electronic pcrformance support systems can help to improve pcrformance. This technique 
will be discussed in more detail later. 

Attentional limitations and working memory limitations also affect performance during the cognitive 
stage. Unlike the expert who can perform many functions simultaneously because some steps have already 
become automatized (converted to procedural knowledge), the novice in this stage must actively rehearse the 
information concerning the problem in working memory while also attending to other incoming stimuli. 
Working memory represents the working space in which thinking and decision making takes place. Because it 
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has a hmited-capacity >o hold information temporarily for processing (Miller, 1956) and be actively involved in 
processing information, the more stimuli that it muii attend to the less resources it has available for carrying 
out die task. The more competition between processing resources the longer it will take to generate a solution 
it at all. As the customer service representative (CSR) progresses into the associative and automatization 
stages of development, more and more of these resources are freed up because of the transition from controlled 
to automatic processing. 

In the cognitive stage, much of the processing that is done is considered to be controlled processing 
This type of processing is characterized as a slow, generally serial, effortful, capacity limited, subject controlled 
processing mode that must be used to deal with novel or inconsistent information compared to automatic 
processing which is fast, parallel, effortless, not limited by working memory capacity, not under direct subie-t 
control, and performs well developed skilled behaviors (Schneider & Shiffrin, 1977; Schneider & Fisk 1983) 
Informauon relevant to the task must be rehearsed to keep it from decaying while at the same lime control 
mechanisms are operating in working memory to search for a solution. 

Two aspects of attention theory (Schneider & Shiffrin, 1977; Shiffrin & Schneider, 1977) are 
operating in this situation: (1) distraction; and (2) limited-capacity. Distraction is the result of divided attention 
due to attempts at parallel processing. During controlled processing this can impede performance because the 
novice CSR does not have the repertoire of automatic skills that the expert CSR has developed in the 
automauzation phase. Chase & Simon (1973) have witnessed this in experiments with novice and expert chess 
players. The expert chess players are able to focus on higher-level skills than the novice due to superior 
memory for sequence of moves and memory for familiar patterns. " 

This limited-capacity aspect is implicit in the explanation of distraction. If only a limited amount of 
resources can be utilized effectively at one time, then the more a CSR can accomplish with automatic 
processing, the more he\she will have available for memory consumptive controlled processing This 

foTStoTrforn^ Pr0CCSSing hdPS l ° al,eVialC "» proCCSSi " g ,imitations Characteristic 

The Associative Stage 

The second stage of the development of skilled performance is the associative stage. In this stage 
declarative knowledge is compiled into sets of procedural knowledge called productions. Anderson (1987) refers 
to this process i as ; knowledge compilation. This is the crucial step in developing domain-specific skill 
According to Anderson, the learner uses what he calls weak problem-solving methods to generate rjroduclions 
S^T iS*^ , W f k 1 methods inc,u <fc *e use of analogy, means-ends analysis, m^XS!iSSS^ 
^S^aHiS ? '"f 1 ™ lhe , decbra,iw knowledge and give it meaning in a performance 

context. The result is a set of productions for solving problems in a particular domain. 

Productions are units of knowledge in the form of conditional statements, such as IF-THEN 

^ C nf "^StrfT^,? 6 c ° ndlUons ^ uired for a Parties type of action. In Anderson's ACT* model this 
type of knowledge would make up the memory component called production memory. During this stage errors 

n rST?,^ "* a " d e '" ninated making Potions more efficient in Sng 

.« . L informaUon ,n work ing memory must match a production in the production memory for it 

to trigger a production response. In the case of the CSR learning how to handle a common custon5?rcque° 
such as converting a document from an old software version to a new version, the CSR using a pSS 
system (a group of productions) for mis problem can now concentrate on problem solving straEand 

P f ° f ^ com ? mon P rocess is elimination of theneed to Sid £££& ^formation 

nr^S"' ; } " • S 0(58 " 0t have to spend P rocessin g resources trying to remember each step of the 
££2 ^ e *PC" ence declarative mterference between performing the skill and remembering similar 

The production for each skill has a condition that specifies the circumstances under which the 

SHnT T'n" T 1 ? acU0 " ^ specifies what shou, ^ e done whc " the producS^pTta (Anderson 
1982). Both controlled and automatic processing are still required in the associative stage to solve problems 
The more familiar die CSR becomes with principles and concepts in this domain, die more a^ioS an? 
cogn uve resources he or she will have available to use on developing automatic processing SsSS 
K^ r ^ S ^ C . 0f H dWC,( * mcm shows incrcases in PcrformanL due to a t^S^^mZSM 

SS^ ,,y proccssing 10 a morc aulomalic ' parallcl - ciL > p ™ g SSL* 
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The Autonomous Stage 

The third stage of skilled performance acquisition is the autonomous stage. In this stage automatic 
processing has developed for much of the skill These automatic productions are shown to be relatively 
resource free, modular, fast, accurate, coordinated, not under direct control, and show high transfer (Schneider & 
Fisk, 1983). Now that much of the skill can be carried out without using working memory resources, 
controlled processing can be utilized to maintain strategy information, results of previous actions, and time 
varying information. Many of the problems associated with working memory limitations can be overcome as 
automatic processing becomes available. 

The development of automatic processing and its relationship with controlled processing has been 
studied to determine the mechanisms of skilled performance (Schneider & Fisk, 1983). Automatic processes 
arc developed only through extensive, consistent practice. Consistent practice occurs when the stimuli and 
responses are consistently mapped (CM), that is, across training trials the individual makes invariant overt (or 
covert) responses to stimuli (or classes of stimuli) (Schneider & Fisk, 1983). Research conditions where 
stimuli are not consistently mapped, referred to as varied mapping (VM), the subjects' performance did not 
improve even after thousands of trials. In detection paradigms, research has shown that extended practice at 
consistently attending to a subset of stimuli results in quantitative and qualitative changes in performance 
(Moray, 1975; Schneider & Shiffrin, 1977). Reaction times are faster, detection performance is less affected by 
mcmoiy load or number of channels, and performance becomes much less sensitive to attentional resource 
demands (Schneider & Fisk, 1982). This suggests that dual-task performance can be improved with CM 
training in some of the task components. 

Schneider and Fisk (1983) outline a series of assumptions about automatic and controlled processing 
concerning the mechanisms involved in acquiring skilled performance: 

1. Practice leads to the development of a large vocabulary of automatic productions which perform consistent 
stimulus to response transformation; 

2. Practice makes automatic productions resource free, autonomous, fast, accurate, and coordinated; 

3. Changing the contents of working memory can change the enabling (test) conditions that switch in different 
sets of productions appropriate for a given situation; 

4. Practice can incorporate both internal and external context cues to enable appropriate sets of production; 
Both internal and external stimulus cues can cause a strategy shift which is then maintained in working 
memory; 

5. Practice improves chunking of information about the outputs, goal stales, and inputs of the situation. This 
chunking allows very detailed information about rapidly changing events to be maintained in a very limited 
working memory. 

These mechanisms help distinguish automatic processing from controlled processing and further delineate their 
key roles in the development and maintenance of skilled performance. The rest of this paper will focus on the 
development of electronic performance support systems that effectively mediate between these competing 
mechanisms. 



Performance Limitation #1 - Long-term Memory Demands 

Long-term memory capacity is considered to be unlimited, although this would be very difficult to 
prove. The question usually of concern for increasing long-term memory performance is in providing the 
correct retrieval cue (Tulving & Thompson, 1973). Finding an appropriate retrieval cue to activate knowledge 
in long-term memory is tied to the context in which the information was originally processed in working 
memory (Shiffrin, 1976; Tulving & Thompson, 1973). 

Another possible explanation of retrieval failure is that the level of processing was weak during the 
initial presentation. This could result in information never making it to long-term memory, stored too weak to 
access, or decayed due to insufficient processing (Craik & Lockhart, 1972;). This theory is an alternative to the 
information processing model previously discussed. No matter what the explanation is for long-term memo * 
retrieval failure, there appears to be a limit to what can be expected of a trainee f n a performance context. 

The performance problem for consideration here is of the CSR being unable to store a vast amount of 
information prcsv ited to himMicr in the training session. Because the trainee is involved in controlled 
processing during the training session, he\she is limited to the amount of information hcNshe can sufficiently 
process at one time (Navon & Gopher, 1979; Schneider & Shiffrin, 1977). There are many factors that can 
affect this including prior knowledge, domain complexity, amount and rate of stimuli presented, and the 
amount of practice available in the training session. The instructional designer of the training program may be 
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for some of ***** problems, but realistically, there is too much declarative knowledge 
involved m learning a software package to be acquired in a traditional training program Know,cd SC 

Performance Solution #1 - Provide An Extension Of Long-Term Memory 

arpf0 . 0ne w .ee*jnt° the job the CSR receives a call from a customer who needs to know which printer files 

t^SSS^S^ Printen WhUe ^ fi,CS Was a Covered": the SinTng 
session, the C^R did not have the opportunity to practice this skill using the customer's printer as an examnir- 
This is an example of miss ng declarative knowledge from long-term memory. The SSTSf' 

wtth another menu screen. This screen has a list of frequendy asked questions atouTpr iSl She eS 
S„°f h q rco 0 " heNshe K nceds - "^"8 User Printer FUes", and clicks on the button U poS promm 
asking the CSR to input the name of the user's printer. After a few key strokes or mouse a new saeen 
comes up that lists the required files for the user's printer. The CSR reads off the fSS^te^ST 
sends her on her way all in a matter of a few minutes. 

What happens in this situation is that the CSR is using the EPSS as an extension of his\her in™ 
Ern'H—^ n °} haVC * St ° re MS W* ^ infrequendy used infolS^^ 
^nH k " Wl f J ° b *, dS indicate 1,131 ^ can brid S c 1,10 mem °ry associated leS™ that 
expand with die complexity of equipment (S wezey & Pearlstein. 1974; Rowan. 1973;13e WT^vSu 

Sn^S . , dCVe,0P *^"^t Producdons (procedural knowledge) for using the EPSS » Sly ^access die 
r™Th ls /^onable to expect this skill to develop from controlled processing o S£S£^e£n> 

&Ssf iSff £ES* (Sch " eider , & Sh u iffrin - 1977 = Shiff ri" & Schneider. 1977; AnSTSS 
& Fisk 1983). The process used to solve the problem can be consistendy mapped because wh k there are 
many different printers available, there is one general procedure for accessingTe faKStf ^ngXTpSS 
i, v.i If , ° f a - PPr0aCh l ° 1116 * wto P"«>i of skilled performance has an explicit transrer qualkv about 
Ln^ t™"^™ «s na spent on acquiring large amounts of technical infSSStoTSB ^me if 
spent on genera problem solving skills that have a high transfer quality. DcvclopinKTton^uc nioSne 
skills for retrieving information from the EPSS's infobase is wha? is focused oSKS 
i£?rT !" ° f Pri ° r kn0wledge ***** about »» «>^are and h^w^oScSR to 

nTvHnn !y ^ b "r*°" ? r ° b,CmS usin S ** EPSS - ^iples and concepts J^Sdh^«SL the CSR 
m developing problem solving strategies and heuristics should be emphasized (Ande^on 1987) 

retrieval Fo7exl°ole i?L*r$R h° 5° U,d 3150 "T" ^ ° f ^ s » «*» in^Semory 
retrieval. For example, if the CSR had encounter this same exact problem two months prior to this but could 

not remember the missing declaradve knowledge, the EPSS could provide missing SJaTcues Tte CSR 

may remember by just re-visiting a couple of the screens in the previous searc EvenTf the CSR exTrml , 

enure search process, the exercise will strengthen the auio m ^ P p ro ce«iSSp2S , l oftiSl 

Performance Limitation #2 - Infrequently Used Skills 
with a skShSSSK* fSST^T^ ^ individuals eXlensive - consistent pracdee 

consistency (Schneider & Fisk, 1982) i^ute, ana uegrce 01 

incompetency is not the trait that die software manufacturer wan* to coiwey ? Ktome« 
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Performance Solution #2 - An On-Line Coach For Infrequent Tasks 

The advisory component of the EPSS guides the user by providing interactive advice on how to solve 
problems, develop strategies, and in making decisions. This is the component of the EPSS most directly 
responsible for helping novices function like experts during infrequently performed tasks. An example using 
the CSR will help demonstrate this aspect. 

The representative has just received a call from a user wanting to know how to create parallel columns 
in a word processing document. The representative is aware that this procedure is possible but has only 
performed it once six months ago in the training session. Normally, this would be a time to either drag out 
the reference manual and start searching for a listing of the procedure, or start calling other CSR's to find 
someone who knows how to perform this task. Either one of these alternatives will probably result in a time 
consuming process that would be perceived poorly by the customer. 

With the help of the representative's EPSS, heNshe can quickly access an on-line coach that has been 
linked to the infobase. By clicking on the button for import and then the button for coach, the representative 
receives prompts for information concerning the problem. Alter inputting the two types of documents and 
their manufacturer's names, the coach begins to take the representative through a step-by-step process using 
graphics, text, and audio segments. The representative can quickly run through the process in a few minutes 
while the customer is on hold. Then heNshe can report back to the customer with the necessary knowledge to 
act like an expert 

A list of important questions may also be provided to the representative to ask the user before 
initiating the procedure. By inputting the customer's answers into the system, the on-line coach can provide 
specific tips and pointers about the customer's software or system, as well as cautions for avoiding errors. The 
customer does not have to know the source of this representative's demonstrated expertise. 

Again, the point to be made here is that the representative is able to function as an expert with the 
training of a novice. The learning that is taking place is two-fold: (1) skill development in using the EPSS to 
solve problems and; (2) incidental learning of the procedures involved in solving the problem for the customer. 
As the representative's experience in using the EPSS increases, heNshe will begin to build a repertoire of 
productions for efficiently accessing problem solving solutions built into the system. While there is currently 
no specific research to back up this claim, it can be hypothesized from other experiments using job aids. 

Stone and Hutson (1984) investigated user selection patterns and strategies of information while using 
a hypertext/graphics computer-based job aiding system to perform an assembly task. The researchers found a 
5% error rate with subjects using the hypertext system compared to a 100% error rate in previous studies using 
text only, a 91% error rate with studies using graphical information only, and a 75% error rate in studies using 
both text and graphics but without the rapid and organized access provided by the hypertext structure. Because 
the subjects (N = 20) varied in their preferences and repetitions in selecting information, the researchers 
concluded that when people can get the information they need when they need it, performance on procedural 
tasks is facilitated. But since people vary in the information they feel they need, it may be useful to have more 
information (and more forms of information) available than any one person is likely to need. This is 
accomplished while keeping the surface text uncluttered in order to avoid information overload for those who do 
not need that information. Booher (1978) cites Guillford (1973) as he states that considerable evidence has been 
accumulated in education and training research to show the advantages of using a variety of media in work 
environments to help workers meet performance objectives. 

Incidental learning from using an electronic performance support system would be determined by the 
amount of controlled processing resources invested in solving the problem, as well as many other factors. 
According to Navon and Gopher (1979), for a given individual at a certain moment, a task is characterized by 
several parameters, such as sensory quality of stimuli, predictability of stimuli, availability and completeness 
of relevant memory codes, stimulus-response compatibility, response complexity, and amount of practice. By 
increasing the amount of resources invested in a task, the learner can increase the performance gain. However, 
since the purpose of the EPSS is to reduce the amount of resources invested in a task, it can be assumed that 
incidental learning will vary from task to task according to the factors previously mentioned. 

Performance Limitation #3 - Overload of Working Memory 

Human decision making tasks, such as those involved in command and control, require that a large 
amount of information be simultaneously considered and integrated into an appropriate decision (Klapp & 
Philipoff, 1983). Because working memory is considered to be limited in its capacity to hold information for 
processing (seven, plus or minus two items; Miller, 19S6) and in its ability to process information while 
attending to other stimuli (Peterson & Peterson, 1959), tasks that require the simultaneous processing of 
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information put a burden on this system. The way in which this burden is accommodated through controlled 
and automauc processing determines performance of the tasks involved. 

An example of this condition using the customer support representative scenario would be a condition 
in which the representauve is required to use controlled processing resources to search for a solution to a 
customer s problem while also responding to the customer's questions. This process becomes even more 
aggravated when the representative must also hold several pieces of declarative knowledge in working memory 
while involved in problem solving and decision making. To make matters worse, the declarative knowledge 
could be lost from working memory when a co-worker asks the representative a question about another 
problem. 

The Performance Solution #3 - Reducing Working Memory Load 

The solution to a performance problem created by ah overload of working memory capacity is to create 
an extension of the working memory. This can be accomplished with an EPSS. In this situation the 
representauve has been asked by the customer to trouble-shoot a problem. The problem is that the customer's 
document file will not print on a new type of printer. Ordinarily, the representative would have to first ask the 
customer a series of questions to help in the trouble-shooting process. The answers to these questions would 
have to be maintained in working memory while the representative attempted several problem solving 
strategies. The results of each attempt would also need to be maintained to facilitate the problem solvine 
process. b 

Nhi nnlvSSMoJ? repr ef ntative to"? was desi gn«« for this type of performance problem. 

Not only docs the EPSS provide prompts for the series of questions, it holds the answers in memory cells on 
the screen so that the representauve can access them at any time. The EPSS is assuming the role of the 
working memory in keeping the information active for processing. The advisory system of the EPSS cuides 
the representauve through the trouble shooting phase and keeps record of the progress of each attempt by 
asking the representauve to respond to questions after each step has been executed. 

By assuming these working memory functions, the EPSS frees processing resources so that hc\she 
may effectively communicate with the customer. Other environmental distractions do not impede the progress 
because the EPSS ,s storing the necessary information and results. This level of artificial intelligence may not 
be available with every EPSS. however, it is easy to sec how marginal aucmpts to reduce working memory 
load can increase performance. t> y 

Conclusion 

This new paradigm in performance technology requires much more understanding about cognitive 
Uieory and processing hmitauons in a variety of performance contexts before a useful design theory can be 
developed for electronic performance support systems. Research is needed to determine if actual learning is 
uiking place when the learner is not required to engage in controlled processing while solving a problem 

^nSH 0P °.K ng ? ^ te ^l SOme 0f ** assumptions in this paper in performance environments' 
supported with and without an EPSS. careful observations of people using EPSS's in context may provide the 
necessary insight to develop a well-structured research study. ihuviw. mc 

»„.hn„i„™ ~ Ugh ?* Perf °, rmanCC 'nations described in this paper sound promising from a performance 
technology perspecuve. enthusiasm must be tempered with careful research. The implications of a workforce 
relying too much on a machine for aiding cognition could prove disasuous. Machines, especially computcr- 

SSi^" kn0 , Wn H° " g ° d0Wn '' A1S °- m0Uvati0n 10 P^ 0 ™ ** ■*» •naVbecome depleted if 
it is perceived that the computer has assumed a superior role to the person using it. Nonetheless the 

technology is already being used in this capacity by developers using intuition due to a lack of cognitive theory 

applied to this type of performance intervention technique. Perhaps as the industry support for this technique 

continues to grow, performance technologists and cogniiive psychologists will discover the reasons why 

decu-ome i performance support systems seem to enhance human performance and as a result develop better 

methods for designing, developing, and evaluating them. 
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